Dynatrace

Dynatrace metrics through JMeter to Jenkins

18th September 2014: Download the jmeter script

Put these extensions under jmeter/lib/ext: jmeter extensions.zip

Before downloading, please read our terms and conditions

If you are a performance tester (or devops or system admin etc.) you may be aware of Dynatrace If not, it really is worth investigating. It gives you full application profiling under load with a very small app server footprint. It is brilliant!

So, I want a simple method to get metrics out of dynatrace into our CI builds, to contribute to the pass/fail criteria.

Dynatrace does provide a REST interface and I started looking at the java dll for this (which is used below) but this is not straight forward and at the time of writing does not always give you what you want.

There is also an HTTP REST interface accessed through a browser that can also give up itís data to a simple script. This is the second method employed below to build into a JMeter/Jenkins CI process.

Iíll give the code below but first some screenshots of the sort of data I am after

ScreenShot100b
host health2
gc stats

Dashboards

You need to setup these dashboards for specific data you want to collect in the jmeter scripts.

1: purepaths ordered by response time descending:

purepaths dashboard

CPU and Memory stats for the app server. Note the underlying data in this dashboard is more extensive than shown in this screen shot:

cpu and memory dashboard

An example of some of the data within this system monitor dashboard:

dashboard memory graph inset

The JMeter model and code

In this example, we

1. setup a basic authentication header,
2. use the dynatrace dll to get the purepaths objects from the dashboard shown above,
3. call the HTTP REST interface to get data from the server metrics dashboard shown above and
4. we graph this data.

Steps that will come later will include outputting pass/fail criteria to .jtl files for the Jenkins build screen

The model outline:

dynatrace jmeter script

The PreProcessor and Header Manager elements setup basic authentication as described on this page

The dynatrace library calls then use the dll to get a few metrics first and then get the objects from the purepaths dashboard. The metrics were experimental but Iím leaving the code here incase you want to play around with it. I am mainly after the purepath info for this step.

Note also, I am connecting to the dynatrace server but it looks like it is on the localhost. In fact, I have set up an SSH tunnel with putty to the server to forward ports 2021 (for the client) and 8020 (for the REST interface). This does have implications as Iíll need to sort this out when I move the script to our Jenkins build server. Ports are typically locked down but Iíll look into this later.

So here is my code. Note my purepaths dashboard is already setup to list the purepaths in descending order. So I can easily find the 95th percentile of the response times listed. Note also I am using SampleResult.setResponseData to output my Ďreportí for other jmeter modules to work with:

import com.dynatrace.diagnostics.automation.rest.sdk.RESTEndpoint;
import com.dynatrace.diagnostics.automation.rest.sdk.entity.Agent;
import com.dynatrace.diagnostics.automation.rest.sdk.entity.ServerStatistics;
import com.dynatrace.diagnostics.automation.rest.sdk.entity.BaseRecord;
import com.dynatrace.diagnostics.automation.rest.sdk.entity.Record;
import com.dynatrace.diagnostics.automation.rest.sdk.entity.PurePath;


output = "";

// Start and Stop Session Recording
RESTEndpoint endPoint = new RESTEndpoint('admin', 'password', 'http://localhost:8020');

String serverVersion = endPoint.getServerVersion();
log.info("serverVersion: "+ serverVersion);
output = output + "serverVersion: "+ serverVersion + "\n"

ServerStatistics stats = endPoint.getServerStatistics()
long events = stats.getReceivedEvents()
log.info("events: " + events.toString())
output = output + "EV-events: " + events.toString() + "\n"

long HeapMemoryUsed = stats.getHeapMemoryUsed()
log.info("HeapMemoryUsed: " + HeapMemoryUsed.toString())
output = output + "HMU-HeapMemoryUsed: " + HeapMemoryUsed.toString() + "\n"

//String sessionName = endPoint.startRecording("easyTravel", "Session1", "My Load Test Session", "all", true, false);
//System.out.println("Started recording Session: "+ sessionName);
//endPoint.stopRecording("easyTravel");
 
       // list all connected agents
       ArrayList<Agent> agents = endPoint.getAgents();
       for(Agent agent : agents) {
           String name = agent.getName();
           System.out.println("Agent Name: " + name);

           output = output + "Agent Name: " + name + "\n";

           if(name.equals("bips"))            {
               double totalCpuTime = agent.getTotalCpuTime();
               String systemProfile = agent.getSystemProfile();
               String host = agent.getHost();
               int purePathCount = agent.getTotalPurePathCount();
           }
       }

       java.util.ArrayList<? extends Record> record = endPoint.getObjects("bips purepaths 5mins","bips");
       int purePathArrayCount = record.size();

       if(purePathArrayCount > 3) {
           int ninetyfifthIndex = (int) (purePathArrayCount * 0.05); //this is the 95th percentile as dashboard is ordered
           Record ninetyfifthPP = record.get(ninetyfifthIndex);
           double ninetyfifth = ((PurePath) ninetyfifthPP).getExec();
           output = output + "NF-95th purePath exec time: " + String.valueOf(ninetyfifth) + "\n";
       }
      
SampleResult.setResponseData(output,"UTF-8")

 

Example output from this code:

library calls output

And the page data extractor can then graph these values (the output was formatted with this in mind):

dynatrace purepaths 95th jmeter graph

The server metrics section of the script uses a different approach. Instead of the dll, it uses the HTTP interface. The request is as follows:

server metrics request

This gives output such as this:

server metrics output

And then we use a couple of data extractors together with some code and a another jp@gc page data extractor to graph the numbers we want:

extract used memory extract cpu idle

report metrics code (inside a JSR223 Sampler):


output = "";

//log.info('memory numbers')
//log.info(vars.get('usedMemoryRaw'))

usedMemoryRaw = vars.get('usedMemoryRaw')

last = 0;
start = 0;
while(start >= 0) {
   start= usedMemoryRaw.indexOf('avg=',last+1)
   if(start >= 0)
       last = start
}

end = usedMemoryRaw.indexOf("\"",last+6)

usedMemory = usedMemoryRaw.substring(last+5,end)
log.info('used memory average to report = ' + usedMemory)
output = output + "ME-MemUsed: " + usedMemory.toString() + "\n"

//log.info('cpu numbers')
//log.info(vars.get('cpuIdleRaw'))

cpuIdleRaw = vars.get('cpuIdleRaw')
last = 0;
start = 0;
while(start >= 0) {
   start= cpuIdleRaw.indexOf('avg=',last+1)
   if(start >= 0)
       last = start
}

end = cpuIdleRaw.indexOf("\"",last+6)

cpuIdle = cpuIdleRaw.substring(last+5,end)
log.info('cpu Idle average to report = ' + cpuIdle)
output = output + "CP-cpu Idle: " + cpuIdle.toString() + "\n"

SampleResult.setResponseData(output,"UTF-8")
 

Output from this report code:

report metrics output

And the page data extractors used to graph these things:

dynatrace mem used jemter graph
dynatrace cpu idle jmeter graph

So this is all working thus far. As stated above, the next step is to build .jtl files from this data for CI pass/fail. I guess I will be handling this like my standard response assertions, collecting these graphs into csv files on my cloud injectors, from there calculating an overall 95th percentile figure for each metric, and then setting some limits for CI.

The first thing to consider for the next step is to run these steps ONLY for one thread on one cloud injector. We only need to gather these gross stats at that level. I do have code on this page to do that bit

[Home] [About (CV)] [Contact Us] [JMeter Cloud] [webPageTest] [_64 images] [asset moniitor] [Linux Monitor] [Splunk ETL] [Splunk API] [AWS bash] [LR Rules OK] [LR Slave] [LR CI Graphs] [LoadRunner CI] [LR CI Variables] [LR Bamboo] [LR Methods] [LR CI BASH] [Bash methods] [Jenkins V2] [Streaming vid] [How fast] [Finding Issues] [Reporting] [Hand over] [VB Scripts] [JMeter tips] [JMeter RAW] [Dynatrace] [Documents] [FAQ] [Legal]

In the Cartesian Elements Ltd group of companies