LR CI Graphs

Getting LoadRunner CI trend graphs into Jenkins

[Read below about files in this download] 18th June 2015: Download this project

[See notes below] 12th Feb 2015: HP Jenkins Plugin

Before downloading, please read our terms and conditions

I’ve built a LoadRunner Jenkins slave and am running CI tests using it. Everything is fine and LoadRunner gives us a pass/fail based on SLAs but really, we’d like to see trend graphs on the Jenkins front end rather than just pass/fail.

So I’ve built a shell script that does just that. This script currently works for the percentile SLAs, which are the main ones we want to graph, but at a later date will investigate other metrics.

jenkins_graphs_single_example

Implementation details

When you set up response time SLAs for your LoadRunner CI scenario, you see these two screens:

image00104
image00204

And the results xml file (which is copied over to the Jenkins workspace), reflects the goals and achieved values for these SLAs:

Starting scenario...
======================================================================== ====
Analyzing results...
Analysis Results:

SLA report:
================================
Full Name : Transaction Response Time (Percentile)
Measurement : PercentileTRT
Goal Value : 0.5
Actual value : 0.131
status : Passed
================================
================================
Full Name : Transaction Response Time (Percentile)
Measurement : PercentileTRT
Goal Value : 0.8
Actual value : 0.271
status : Passed
================================
================================

So if we’re lucky (and it turns out we are), we can use this xml file to build some Jmeter jtl files which the Jmeter Jenkins plugin can then pick up and graph.

To do this, we need to add some extra steps to our Jenkins build and, as it turns out, add a slight delay for the xml file to be copied over (this may need increasing on your sytem, I haven’t run this on a full project yet)

Steps in the Jenkins build:

Only run this build on the bespoke LoadRunner slave:

image00303

Build steps:

  1. [update: code for tidying up the workspace is now shown. It is important for the graphing to at least delete the old jtl files before each test run]
  2. The printenv command first prints out our environment status for the record
  3. Then the lucky part: Run the scenario with the HP Jenkins plugin. Luckily, this actually puts the xml results file into the workspace after it has finished, I guess ready for the HP report step to pick it up. But this means we can intervene now and:
  4. Run our specially designed jtl building shell script. I tried running this direct in the jenkins shell but it wasn’t happy. Something was lost in translation with the grep step. But it works fine on the LoadRunner slave.
  5. NOTE: Transaction names are taken from the SLA screen above. Results are ordered alphabetically so you may want to add step numbers to the front to get the order you want.
  6. I’ve added the disk space monitor to this project. This keeps an eye on the slave disk space and graphs percentage used in the Jenkins front end. Again, you may want to change the step name to place this graph where you want it.
build_steps

Post build steps then follow:

  1. Run the HP test reports. This step still picks up everything, including error rates and any other SLA you have set and will set the build flag to fail if required.
  2. Run the JMeter plugin reports. This now picks up our bespoke jtl files and produce the trend graphs (see below).
  3. (Further investigation will be carried out shortly about graphing other SLAs with the JMeter plugin. Watch this space.)
image00502

Jenkins output

The immediate Jenkins build page shows the HP plugin output with the standard pass/fail graph on the right:

image00703

But clicking on the ‘Performance Trend’ link now shows the trend graphs from the LoadRunner xml results file. And the script checks for pass/fail against the goals and draws the error graph as well. [update: the disk space monitor is now included in this project]:

jenkins_graphs_full_screen_example

The shell script itself is available for download at the top of this page. Note, this script needs to be placed in the Jenkins workspace on the LoadRunner slave. And it needs permissions set to 777 in a cygwin terminal and in windows, permissions set to everything for everybody (I think if you just do the windows step, that might be enough). [update: do this for the disc_space script as well, included in the zip file].

NOTE: if you do change this script, make sure line endings are linux style, with just \n on the end rather than \r\n like windows. Otherwise, bash will have some problems running it.

NOTE ALSO: There may be other Jenkins steps that your project needs here, such as archiving results files. This page does not show a real project at this stage, just an introduction to the trend graphing solution.

AND: Don’t forget (note for me), we also need a script to manage disc space on the LoadRunner slave and keep it under control. My AWS server is quickly running out of disc space. LoadRunner can be quite prolific with its output files. I’ll look into this next as I’ll need it for running any enterprise level test runs. I am now clearing the workspace each run, which will help. I’ll see where the real disc space issues lie when I run bigger projects.

More notes for later use:

I will be planning on copying over the HTML analysis folder to the workspace on jenkins. This is currently stored in the user temp folder. In my case: C:\Users\Administrator\AppData\Local\Temp\36fb71*

*To find the correct folder name for the last test run, look in the analysis.log file under the local\temp directory. Look for this line:

    Notify: The version of the .eve file "C:\Users\Administrator\AppData\Local\Temp\36fb70\LRR\localhost_1.eve" is: 3.

**NOTE: this relies on the LR Jenkins plugin working I think. The above is actually wrong. Keep an eye on this. I’ll update this page later. HP are working on the plugin right now.

You may also want to copy over other details from the analysis folder, just to make click through reporting easier within Jenkins. One other thing you can do is run the analysis launcher that the plugin uses to create the final results xml file.

Run this as follows:

To launch the tool manually you can open a CMD window, navigate to “C:\Program Files (x86)\Jenkins\jobs\loadrunner bips local\workspace” and then run:

".\LRAnalysisLauncher.exe" C:\Users\Administrator\AppData\Local\Temp\36fb70\LRR\LRR.lrr C:\Users\Administrator\AppData\Local\Temp\36fb70\LRA\LRA.lra C:\Users\Administrator\AppData\Local\Temp\36fb70\HTML\HTML.html

Replace 36fb70 with whatever the name of the temp dir where the LRR files are.

And the LoadRunner files needed in the workspace are included here for convenience. Note these files are all available as part of the LoadRunner install and you may want to get them from your own installation as no doubt these versions will go out of date. Take heed: hpTools_02Dec2014.zip

End of January 2015

Well, I’ve been working with HP support for more than six months now but finally, we have a fully working Jenkins CI plugin (see the bottom of this page for the struggles we have been having: LR slave). They can’t tell me when it will get into the offical release but here it is if you want to run full CI tests with Loadrunner 12:

    HP Support LR Jenkins CI plugin - SEE TOP OF PAGE FOR DOWNLOAD LINK

Date of posting this software is shown at the otp of the page so bear this in mind if you want to use it - check through Jenkins whether there is a newer version available.

and I have now added an extra component to the CI project (link at top of page), a shell script which uses this version of the plugin to also copy the html report straight to the Jenkins workspace (and build archive) to allow click through  from the build page.

The Jenkins project to use these scripts looks like:

1. execute a shell to tidy up the workspace:

    #!c:/cygwin/bin/bash (NOTE: we found this is needed on a linux Master Jenkins for sure)
    #tidy up the workspace
    echo tiding up workspace: ${WORKSPACE}
    echo $PATH
    PATH=/usr/bin:/cygdrive/c/cygwin/bin:$PATH
    echo $PATH
    cd "${WORKSPACE}"
    rm --interactive=never -f *.xml
    rm --interactive=never -f *.jtl
    rm --interactive=never -f *.txt
    rm --interactive=never -f report_location

    #tidy up the workspace - NOTE: ALL OF THEM! - controller SHOULD be limited to one project running at once -
    echo full tidy up workspace: ${WORKSPACE}
    cd "${WORKSPACE}"
    echo workspace =
    pwd
    ls
    echo try finding directories
    if [ ! -d "dir_temp" ]; then
      mkdir dir_temp
    fi

    /cygdrive/c/cygwin/bin/find . -name "*" -type d | grep / > dir_list
    echo "now delete directories"
    if [[ -s dir_list ]] ; then
    echo "dir list file has data - deleting directories now"
    rm -rf $(/cygdrive/c/cygwin/bin/find . -name "*" -type d | grep /)
    echo "deleted directories"
    fi
    ls
    echo done
    printenv

2. Run the load test scenario using the HP plugin

3. Execute the graphing shell scripts (currently like this):

    #!c:/cygwin/bin/bash (NOTE: we found this is needed on a linux Master Jenkins for sure)
    echo $PATH
    PATH=/usr/bin:/cygdrive/c/cygwin/bin:$PATH
    echo $PATH
    cd "${WORKSPACE}"
    ./sla_to_jtl.sh
    ./report.sh
    stepname="27." ./disk_space.sh

4. Archive the artifacts with the following entries (this step now picks up the html report which was copied over to the workspace by report.sh above):

    *.*,htmlReport/**/*

5. Publish performance test result (using the jmeter jenkins plugin) witht he following entry (I do set thresholds of 5% and 10% but in practice any failures are reported as 100% anyway. This is fine for my purposes.):

    **/*.jtl

     

Watch this space for further updates..

23rd Feb 2015
I have updated the latest version of the plugin from HP above (12th feb 2015). I have removed ‘jtl_script.sh’ and replaced it with ‘sla_to_jtl.sh’, which currently works for average transactions per second and percentiles. I have not yet updated the report script to work with this version so there may be issues. I am looking at this now.

6th March 2015
update to ‘sla_to_jtl.sh’ to include errors per second and error count. NOTE I am not sure that the error count from the plugin is correct but it may still be worth graphing. I have raised this with HP today.

10th March 2015
Updated ‘sla_to_jtl.sh’ to do all the SLAs and added ‘total number of errors’ as a separate graph to ‘error count’. Still trying to find out exactly what these error counts refer to.

10th April 2015
Updated sla_to_jtl.sh to output a quick report csv file, building up the metrics in a file for use with excel or other graphing tools

18th June 2015
Updated sla_to_jtl.sh to allow for when we have no data for a particular request group. Set this to Pass rather than Fail. NOTE: you may want to change this behaviour for your projects.

More problems with a linux master and a windows slave (note the use of #!c:/cygwin/bin/bash above)

Another issue has come up with using our main Jenkins master which is linux with this LoadRunner (windows) setup:

The workspace folder is getting muddled up. Instead of the linux master setup with the workspace folder as:

z/jenkins/workspace/all4_check_urls_15mins (as done in the windows master),

it needs to be: /cygdrive/z/jenkins/workspace/all4_check_urls_15mins and is then trying to save to:

C:\cygdrive\z\jenkins\workspace\all4_check_urls_15mins\cd3eb4

To get around this, I have done the following:

    1. Go to (create if needed) c:\cygdrive\z

    2. From there: mklink /D jenkins z:\jenkins

This will redirect the linux master to the correct location and the windows master will be unaffected. Build this into the LoadRunner AWS image.

Just spotted another issue
For Jenkins to work away happily, you need to check the box ‘automatically create a results directory for each test run’. This may be due to this particular version of the plugin but without this set, it looks in the wrong place for the .lrr results file:

ScreenShot84402

[Home] [About (CV)] [Contact Us] [JMeter Cloud] [webPageTest] [_64 images] [asset moniitor] [Linux Monitor] [Splunk ETL] [Splunk API] [AWS bash] [LR Rules OK] [LR Slave] [LR CI Graphs] [LoadRunner CI] [LR CI Variables] [LR Bamboo] [LR Methods] [LR CI BASH] [Bash methods] [Jenkins V2] [Streaming vid] [How fast] [Finding Issues] [Reporting] [Hand over] [VB Scripts] [JMeter tips] [JMeter RAW] [Dynatrace] [Documents] [FAQ] [Legal]

In the Cartesian Elements Ltd group of companies