Splunk API

Splunk API for trending data the GUI struggles with

We have a simple request from the management: show us the peaks of login and registration into one of our systems so we can see how often we go above certain thresholds. Sounds simple and just the sort of thing Splunk is aimed at. Write a search and timechart the results.

But as always it turned out to be a bit more tricky, so here is my solution. I do make use of jmeter and jenkins, mainly ‘cos these are my day-to-day tools but also, it is good to offer a webpage to the managers that they can view from anywhere. Typical output is:

image006

Those graph look pretty simple but the devil is in the detail. We want the peak per second numbers over a long period of time. We want to see how this peak per second trends and how often it goes high.

The problem is that Splunk indexes tons of data - these figures come from multiples servers and login/registration is just one aspect of the applications. So if you try to get per second charts in the Splunk GUI, you soon hit the 50,000 record limit in searches. The best I could do was over 4 hours.

My solution is to use the API to call my search every hour, looking for the peak number over the past hour, which Splunk can handle, and graph the results with Jenkins. As an aside I write a csv file with the peak time stamp (from the log file) and value on each call. So we have a record for the archive.

I use a jmeter script to call the api and build an output file for jenkins to graph:

image007

Before downloading, please read our terms and conditions

Put these extensions under jmeter/lib/ext: jmeter extensions.zip

Note: to use the API with the free version of Splunk you need to add this setting to $SPLUNK_HOME/etc/system/local/server.conf and restart:

[general]
allowRemoteLogin = always

And in fact you don’t need to login explicitly with the free version so potentially you can do without the session headers.

Here is the peak login script used for the results above: SplunkloginsPeak.zip* (the registration script is similar)
[*updated 12:18am 10th Oct 2013]

NOTE: the search term used in the script is url encoded. You can use this online facility for this: http://meyerweb.com/eric/tools/dencoder/ and the developer does say you can use this locally so if you have sensitive data it could be useful.

The csv file created by the script shows how we have obtained the peak per second value for each and every hour of up time. This is much easier to deal with than full graphs of continuous per second numbers. The time stamps here are the actual peak second in the hour - so they do vary. The last csv field is the tps value:

2013-09-26T11:43:44.000+01:00,5
2013-09-26T12:11:58.000+01:00,4
2013-09-26T13:05:30.000+01:00,4
2013-09-26T14:49:37.000+01:00,6
2013-09-26T15:10:48.000+01:00,6
2013-09-26T16:21:16.000+01:00,6
2013-09-26T17:33:51.000+01:00,7
2013-09-26T18:38:48.000+01:00,8
2013-09-26T19:33:51.000+01:00,11
2013-09-26T20:01:46.000+01:00,8
2013-09-26T21:29:49.000+01:00,10
2013-09-26T22:28:06.000+01:00,9
2013-09-26T23:02:56.000+01:00,5
2013-09-27T00:37:47.000+01:00,5
2013-09-27T01:07:26.000+01:00,3
2013-09-27T02:24:33.000+01:00,3
2013-09-27T03:53:22.000+01:00,3
2013-09-27T04:02:28.000+01:00,2
2013-09-27T05:05:59.000+01:00,2

[Home] [About (CV)] [Contact Us] [JMeter Cloud] [webPageTest] [_64 images] [asset moniitor] [Linux Monitor] [Splunk ETL] [Splunk API] [AWS bash] [LR Rules OK] [LR Slave] [LR CI Graphs] [LoadRunner CI] [LR CI Variables] [LR Bamboo] [LR Methods] [LR CI BASH] [Bash methods] [Jenkins V2] [Streaming vid] [How fast] [Finding Issues] [Reporting] [Hand over] [VB Scripts] [JMeter tips] [JMeter RAW] [Dynatrace] [Documents] [FAQ] [Legal]

In the Cartesian Elements Ltd group of companies