Symptom

The Splunk saved search configured in the Agent with SID (Splunk job id) results in ERROR: CheckException: Splunk topology failed with message: 400 Client Error: Bad Request for url: message. The StackState log in /var/log/stackstate/collector.log shows the following:

2019-08-06 15:50:41 CEST | ERROR | dd.collector | checks.splunk_topology(__init__.py:1002) | Check 'splunk_topology' instance #0 failed
Traceback (most recent call last):
File "/opt/stackstate-agent/agent/checks/__init__.py", line 985, in run
self.check(copy.deepcopy(instance))
File "/opt/stackstate-agent/agent/checks.d/splunk_topology.py", line 115, in check
all_success &= self._dispatch_and_await_search(instance, saved_searches)
File "/opt/stackstate-agent/agent/checks.d/splunk_topology.py", line 155, in _dispatch_and_await_search
all_success &= self._process_saved_search(sid, saved_search, instance, start_time)
File "/opt/stackstate-agent/agent/checks.d/splunk_topology.py", line 162, in _process_saved_search
responses = self._search(search_id, saved_search, instance)
File "/opt/stackstate-agent/agent/checks.d/splunk_topology.py", line 198, in _search
return instance.splunkHelper.saved_search_results(search_id, saved_search)
File "/opt/stackstate-agent/agent/utils/splunk/splunk_helper.py", line 82, in saved_search_results
response = self._search_chunk(saved_search, search_id, offset, saved_search.batch_size)
File "/opt/stackstate-agent/agent/utils/splunk/splunk_helper.py", line 60, in _search_chunk
response = self._do_get(search_path, saved_search.request_timeout_seconds, self.instance_config.verify_ssl_certificate)
File "/opt/stackstate-agent/agent/utils/splunk/splunk_helper.py", line 136, in _do_get
response.raise_for_status()
File "/opt/stackstate-agent/embedded/lib/python2.7/site-packages/requests/models.py", line 862, in raise_for_status
raise HTTPError(http_error_msg, response=self)
CheckException: Splunk topology failed with message: 400 Client Error: Bad Request for url: https://splunkapidev.tooling.domain.org:8089/services/-/-/search/jobs/srvc_stackstate_dta__nobody__stackstate__RMD58e4feb463ac11e00_at_1565099425_16190_89DA7433-D1EE-4944-9376-2FE48FCA08B6/results?output_mode=json&offset=0&count=1000

Cause

Saved search definition contains an error, or the job id (SID) is not available anymore in Splunk. Jobs in Splunk expire, and they are no longer available from jobs/activity screen and saved search.

Solution

Check the status of the Splunk job using SID in Splunk Activity Screen. To do this, you need to extract the job id from the URL provided in the error message. SID is located in the URL right after /jobs/ - .../search/jobs/{SID}/.... Now you can check this job in Splunk Activity menu.