LR Slave

Building a LoadRunner Jenkins Cloud slave image

This page will go through the process of building a LoadRunner Jenkins slave node. The latest version of LoadRunner finally has the changes that are needed for rudimentary CI testing:

  1. The controller is no longer licensed
  2. Support for Cloud injectors
  3. A bespoke Jenkins plugin for reporting

I say rudimentary because thereís not a lot of control on what you can report through the Jenkins plugin and you donít get nice trend graphs like the JMeter solution shown elsewhere on this site [update: You do now! - see here]. However, you do get to use the very best performance tools for your CI tests and can  leverage your existing test scripts if this is your main working application.

Thereís a lot of detail on this page and no pictures. This information is presented mainly for my use, so I can do the same thing next time. If you step through it carefully, you should end up with a working slave node. I have written this after the event, based on my notes.

Steps required:
We are specifically setting up Jenkins master to slave using SSH rather than a straight windows connection because the windows setup requires several ports open that are not allowed in our enterprise. SSH makes the firewall setup much easier. Also, our main Jenkins server is Linux based and this method allows the slave to work seamlessly with both windows and linux masters.

  1. Build an (AWS) Cloud windows box with LoadRunner installed and working locally with a dummy script. This is the SLAVE.
  2. Install Jenkins and the LoadRunner plugin on the slave
  3. Install Cygwin with SSH included
  4. Do all the user/group ssh setup
  5. Setup the SSH keyfiles and test using putty from your desktop (the temporary master)
  6. Install the java JDK on the slave
  7. Configure the master to slave Jenkins startup
  8. Investigate environment variables needed
  9. Test the final solution: run the scenario on the slave from the master, using the plugin
  10. Extra step: Setup Jenkins to run bash shell scripts and test a more extensive project locally using the CI Graphs scripts

Steps 1 to 3 are all standard.

Steps 4 and 5 (my notes at the time):

setup cygwin ssh. From command prompt do this:

At the prompt, run the following commands:
export CYGWIN='ntsec tty'
chmod +rw /etc/group
chmod +rw /etc/passwd
chmod 0755 /var
ssh-host-config -y
net start sshd

then:
mkpasswd -cl > /etc/password
mkgroup --local > /etc/group
open port 22 on windows firewall

- Putty can then connect with your standard windows users name and password.

To use a keyfile, create a .ssh/authorized_keys file. That file must have restricted permissions:
drwxr-xr-x+ 1 Administrator None   0 Apr 15 12:51 .ssh
-rw------- 1 Administrator None 395 Apr 15 13:00 authorized_keys
(you can use: chmod og-r authorized_keys)

and it should contain the public part of the certificate you want to use:

key file output from putty:
PuTTY-User-Key-File-2: ssh-rsa
Encryption: none
Comment: imported-openssh-key
Public-Lines: 6
AAAAB3NzaC1yc2EAAAADAQABAAABAQCKnC0vSrhV2b35fql2I36cN99QXAez/daw
.
.
.
9kI6teg38bxXrxvEoMKLBpxvDmkN5IHiIBIXSGNjmaWLNulMhdBt
Private-Lines: 14
AAABABo2ku0Aqn/B4AWRCKdI6ybMAjWGUgaeGL3UgR4I4mReSzc31v7Z025e3mvS
.
.
.
WD8Tq1dRn4MZBt05d35gOIxuzVRNVIvFaPSH5+9N1fJ+iA==
Private-MAC: 27ce0159bcc914d32a476c2ae4327e23cb923fb6

entry in authorized_keys (NOTE THIS IS ALL ON ONE LINE):
$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCK............................DmkN5IHiIBIXSGNjmaWLNulMhdBt marksKeyPair

We can now connect with putty using the key file rather than username and  password

Step 6 is standard

Steps 7 (from my notes):

NOTE:
Iím setting up a Ďjenkinsí user here as I believe our corporate Jenkins server will need it. But for this page I ended up not using it and just logging in as ĎAdminisitratorí everywhere. It is left for the user to finish this project by switching users at the end. Of course, it is always best not to use ĎAdministratorí if you can help it.

To use a windows box as a jenkins slave using ssh:
create a new user in windows called 'jenkins'
make sure it is a member of the group 'users'
from the cygwin terminal run: mkpasswd -u jenkins -l
ALSO run:
mkpasswd --local > /etc/passwd
mkgroup --local > /etc/group
you should now be able to login with this username and password from putty.
Add the jenkins user to rdp
login as jenkins
open cygwin terminal:
jenkins@WIN-CVUSNOJGM0Q ~
$ pwd
/home/jenkins
Add the keyfile to authorized_keys as done above.
You should now be able to connect with putty using the certificate and username: jenkins:
jenkins@WIN-CVUSNOJGM0Q ~
$ pwd
/home/jenkins

SHOULD also be able to ssh from a command prompt on the master to the new jenkins box. I had to use a non passphrase version of the key file - use putty keygen (puttygen) to make this:
c:\cygwin\bin\ssh.exe -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -i /cygdrive/c/backup/Graylog_openSSH.ppk jenkins@<server ip address>

Connecting Jenkins Master to Slave:

Run this command from the master (in a cmd window):
c:\cygwin\bin\ssh.exe -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -i /cygdrive/c/backup/Graylog_openSSH.ppk jenkins@<server ip address> java -jar c:/jenkins/slave.jar

and this should give something like:
Warning: Permanently added '<server ip address>' (ECDSA) to the list of known hosts.<===[JENKINS REMOTING CAPACITY]===>rO0ABXNyABpodWRzb24ucmVtb3RpbmcuQ2FwYWJpbGl0eQAAAAAAAAABAgABSgAEbWFza3 hwAAAAAAAAAH4=   ¶¶ ?

For me to run this command to start the jenkins slave (from the master),
I then had to login the jenkins slave service with the Administrator account (service settings).

Select 'Launch slave via execution of command on the Master' option.

The loadrunner plugin isn't working yet though. I get this Jenkins log on the master:

Started by user anonymous
Building remotely on LoadRunner cloud controller in workspace C:\Program Files (x86)\Jenkins\workspace\lr cloud 4id test
[lr cloud 4id test] $ "C:\Program Files (x86)\Jenkins\workspace\lr cloud 4id test\HpToolsLauncher.exe" -paramfile props16042014095532293.txt
"Started..."
1 tests found:
C:\Loadrunner\scenarios\Scenario1.lrs
======================================================================= =====
Error: LoadRunner is not installed on WIN-CVUSNOJGM0Q.
Build step 'Execute HP tests from file system' changed build result to ABORTED
Report not found
Finished: ABORTED

Step 8 (from my notes):

Something else I had to do at some point, not sure where it comes in this page:

  1. Create the Workspace folder on the Slave (C:\Program Files (x86)\Jenkins\workspace)
  2. Copy two tools under this directory (HpToolsLauncher.exe and LRAnalysisLauncher.exe). I got these from the master after testing the LR plugin locally.
  3. Create a project file under the workspace folder (lr cloud 4id test)

After much investigation, to get the loadrunner plugin working: First, set the ssh command above to login as Administrator (may or may not be needed)
Then, need to setup the following in .bashrc file for the Admin user:
export PATH=$PATH:'/cygdrive/c/Program Files (x86)/HP/LoadRunner/bin'
export LG_PATH='C:\Program Files (x86)\HP\LoadRunner\'
export ANALYSIS_PATH='C:\Program Files (x86)\HP\LoadRunner\'
export VUGEN_PATH='C:\Program Files (x86)\HP\LoadRunner\'

STILL NOT WORKING!!!
We need to set more environment variables on the JENKINS master. We can see which variables are set on the slave by adding a build step and running 'set'. This prints out the environment in windows. Run the Jenkins job on the slave and use this as a guide to set the master environment variables.

I did try the 'Environment Injector Plugin' in the build but it didn't seem to work for me. Instead
configure the slave node properties on the Master and restart the slave. In the node setup, add the following:
 

NOTE: (March 2015) I am now using the user Ďjenkinsí, after adding them to the Administrators group. In this case, below replace ĎAdministratorí with Ďjenkinsí
ANALYSIS_PATH=C:\Program Files (x86)\HP\LoadRunner\
APPDATA=C:\Users\
Administrator\AppData\Roaming
CommonProgramFiles=C:\Program Files (x86)\Common Files
CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
CommonProgramW6432=C:\Program Files\Common Files
LG_PATH=C:\Program Files (x86)\HP\LoadRunner\
LOCALAPPDATA=C:\Users\
Administrator\AppData\Local
LOG_FILE=C:\Users\
ADMINI~1\AppData\Local\Temp\\ihp_custom_batches.log
LR_PATH=C:\Program Files (x86)\HP\LoadRunner\
PATH=/usr/local/bin:/usr/bin:/cygdrive/c/Windows/system32:/cygdrive/c/Windows:/cygdrive/c/ Windows/System32/Wbem:/cygdrive/c/Windows/System32/WindowsPowerShell/v1.0:/cygdrive/c/Program Files/Amazon/cfn-bootstrap:/cygdrive/c/Program Files (x86)/HP/LoadRunner/strawberry-perl/perl/bin:/bin:/cygdrive/c/Program Files (x86)/HP/LoadRunner/bin
ProgramData=C:\ProgramData
ProgramFiles=C:\Program Files (x86)
ProgramFiles(x86)=C:\Program Files (x86)
ProgramW6432=C:\Program Files
PSModulePath=C:\Windows\system32\WindowsPowerShell\v1.0\Modules\;C:\Program Files (x86)\AWS Tools\PowerShell\
TEMP=/tmp
TMP=/tmp
VUGEN_PATH=C:\Program Files (x86)\HP\LoadRunner\

They may not all be needed but with all of those set on my configuration, it all works and we are home and dry. A build on the Master, configured to run the test on the Slave, now gives:

March 2015 Additional Notes
Things are now working. See
here for more up-to-date details about the plugin. But to work with our main (commercial, linux) Jenkins server, I have set things up slightly differently:

    1. On the slave, add the jenkins user to the Administrators group and then ssh with that user
    2. set the remote working directory to z:\jenkins on the AWS controller. This is due to disc space. Here, I am making use of the AWS large temporary store. The workspace is then under here as well
    3. In practice I have (also) made symbolic links to this location from the C drive. NOTE: you will lose all data from the Z drive when the instance is shut down.
    4. We may need to revisit temporary working space. I have set the scenarios to use z:\temp on the injectors. We may need to consider this on the controller slave as well.

Step 9 (from an actual test run):

Started by user anonymous
[EnvInject] - Loading node environment variables. [Ed: not using this. See above.]
Building remotely on LoadRunner cloud controller in workspace C:\Program Files (x86)\Jenkins\workspace\lr cloud 4id test
[lr cloud 4id test] $ sh -xe C:\Users\Administrator\hudson8058973769114084733.sh
+ printenv
+ sort

[Ed: list of environment variables given here...]

[lr cloud 4id test] $ "C:\Program Files (x86)\Jenkins\workspace\lr cloud 4id test\HpToolsLauncher.exe" -paramfile props22042014101245328.txt
"Started..."
1 tests found:
C:\Loadrunner\scenarios\Scenario1.lrs
======================================================================= =====
22/04/2014 09:12:43 Running: C:\Loadrunner\scenarios\Scenario1.lrs
Cleaning up the environment...
Preparing scenario C:\Loadrunner\scenarios\Scenario1.lrs for execution.
Load Generator localhost connected
Starting scenario...
Scenario C:\Loadrunner\scenarios\Scenario1.lrs ended.
======================================================================= =====
Analyzing results...
Analysis Results:

SLA report:
================================
Full Name : Transaction Response Time (Percentile)
Measurement : PercentileTRT
Goal Value : 0.5
Actual value : 0.249
status : Passed
================================
================================
Full Name : Transaction Response Time (Percentile)
Measurement : PercentileTRT
Goal Value : 0.8
Actual value : 0.434
status : Passed
================================
================================
Full Name : Errors Per Second
Measurement : ErrorsPerSecond
CriteriaMeasurement Value : RunningVusers
LoadThresholds value :
status : Passed
================================

Error(s) summary:
0 ignored , 0 errors

Test PASSED
Test result: Passed
22/04/2014 09:14:25 Test complete: C:\Loadrunner\scenarios\Scenario1.lrs
-------------------------------------------------------------------------------------------------------
Cleaning up the environment...
================================================
Run status: Job succeeded, total tests: 1, succeeded: 1, failures: 0, errors: 0
Passed : C:\Loadrunner\scenarios\Scenario1.lrs
================================================
Report archiving mode is set to: ALWAYS_ARCHIVE_TEST_REPORT
Zipping report folder: C:\Users\Administrator\AppData\Local\Temp\f0e247
Finished: SUCCESS

Step 10: Run a more complete project locally through Jenkins, including CI Graphs

    The main extra piece of work here is to add the cygwin bin directory to the system path so Jenkins can run bash shells. And reboot Jenkins to pick up this change.

  1. Add c:\cygwin\bin and /cygdrive/c/cygwin/bin to the system path (to be on the safe side).
  2. Restart the Jenkins service
  3. Follow these instructions to check that a more complete project can run locally
  4. With a linux Master, we also seem to need #!c:/cygwin/bin/bash at the top of our Ďexecute shellí scripts.

More problems with a linux master and a windows slave

Another issue has come up with using our main Jenkins master which is linux with this LoadRunner (windows) setup:

The workspace folder is getting muddled up. Instead of the linux master setup with the workspace folder as:

z/jenkins/workspace/all4_check_urls_15mins (as done in the windows master),

it needs to be: /cygdrive/z/jenkins/workspace/all4_check_urls_15mins and is then trying to save to:

C:\cygdrive\z\jenkins\workspace\all4_check_urls_15mins\cd3eb4

To get around this, I have done the following:

    1. Go to (or create) c:\cygdrive\z

    2. mklink /D jenkins z:\jenkins

This will redirect the linux master to the correct location and the windows master will be unaffected. Build this into the LoadRunner AWS image.

Finally, I made an AWS image of the Cloud LoadRunner Jenkins slave node.

Also from experience, Keep an eye on the slave disc space. Test runs make lots of files and this needs managing. Best to setup a automated check for this, perhaps use disk_space.sh mentioned on this page (or a similar windows version)

For new projects you will probably need to manually create the project directories on the slave, as briefly mentioned above for this project. 

My plan currently is just to use the slave controller as an injector as well but more could be done here to configure further cloud injectors. My JMeter solution elsewhere on this site includes some details on managing this process. e.g. for sparking up and tearing down AWS images: AWS bash

ADDENDUM (blog for anyone struggling to get this going...)

Iíve really struggled to get a proper scenario going with real load (even just a 10 minute one). For some reason I was not getting the xml results file created in the slave workspace. This stopped the LR analysis proceeding.

I tried getting rid of spaces in the workspace name (by changing the Jenkins build name) and that worked for one run and then failed again.

Tried increasing the time out in the LoadRunner specific settings in the Jenkins plugin - you MUST set the scenario time out large enough to allow your test to run to conclusion.

Tried ignoring errors in the same settings panel.

So far, I can run my proper scenario for 9 minutes including ramp up and down but not 11 minutes. Will carry on investigating...

Finally, It is now working but I canít be clear why (!). It could be that I was not being patient enough with the analysis step finishing. I thought it was hanging and it looked like all the relevant files were not being produced in the workspace. But I may have jumped the gun. I will continue to monitor my CI process with a real project and report back here.

To check this whole setup, I have copied the same scenario and scripts from my Jenkins master to the slave. Set up the same directory structure (C:\backup\Loadrunner\scenarios and C:\backup\Loadrunner\scripts\bips\Stage\). And copied the Jenkins project from the master to the slave (setup the same Jenkins build steps). I DID HAVE TO add the cygwin path to my slave path so I could run the Jenkins bash shells on this box. Iíll add that step above. Add c:\cygwin\bin and /cygdrive/c/cygwin/bin to the system path (to be on the safe side).

I ran the scenario though Jenkins directly on the Master and it worked fine and I repeated this process on the slave. I then found I could run it from master to slave...

It does look like analysing results does take a LONG time with a real project. This could be an issue for CI as time is important for dev team turn around. I guess scaling up the cloud box would help here. This may need further investigation...

Her we go (test output just now):

    Analyzing results...
    Error: Timeout for Analysis passed.
    Exception while trying to analyze results:
    Process must exit before requested information can be determined.
       at System.Diagnostics.Process.EnsureState(State state)
       at System.Diagnostics.Process.get_ExitCode()
       at HpToolsLauncher.TestRunners.PerformanceTestRunner.generateAnalysisReport(TestRunResults runDesc)
       at HpToolsLauncher.TestRunners.PerformanceTestRunner.RunTest(String scenarioPath, String& errorReason, RunCancelledDelegate runCancelled)
    Test result: Error

Iíll try scaling up the AWS instance from a c1.medium to a c3.2xLarge (new generation). Mind you, I probably shouldnít just throw hardware at it as an instant solution. Letís see if this works, can always hone in on the problem properly if it does.

Just seen this (helps to read the manual!):

    Jenkins Plug-in
    Scenario execution timeout (in minutes)
    Ė timeout for lengthy parts of a scenario execution, which include running Vusers, collating results, analyzing results and shutting down the Controller. If any of these operations within any of the selected scenarios is not completed after the specified number of minutes, the step is terminated and declared ďfailedĒ.

So the bigger machine may improve things (make it faster) but the solution should also include increasing the timeout setting in the plugin. I think I may need a faster machine anyway just to keep the CI rolling. Lets see if the new hardware works. I currently have the timeout set to 40 minutes and the scenario runs for 30 minutes. Analysis therefore had in the region of 10 minutes without finishing.

I think that should be sufficient for a CI solution. In fact, ultimately, for our continuous deployment model, I need to do the full performance step within a 15 minute time slot, so something has to change. I will have to aim at a 10 minute intensive test with 5 minutes analysis. Iíll try this later and see what size machine I need to do that.

So far I still havenít managed to get a working test with 20 users for 30 minutes (at 2tps) using the Jenkins plugin. At this point I need to try and get some HP support. Although CI may be aimed at smaller intensive tests, I do also need longer test runs to work for my general day-to-day workload.

Iíll revisit this later. For the moment I will have to carry on with JMeter.

March 2015 Additional Notes
Things are now working a lot better, just about fully there in fact. See
here for more up-to-date details about the plugin etc.

 

[Home] [About (CV)] [Contact Us] [JMeter Cloud] [webPageTest] [_64 images] [asset moniitor] [Linux Monitor] [Splunk ETL] [Splunk API] [AWS bash] [LR Rules OK] [LR Slave] [LR CI Graphs] [LoadRunner CI] [LR CI Variables] [LR Bamboo] [LR Methods] [LR CI BASH] [Bash methods] [Jenkins V2] [Streaming vid] [How fast] [Finding Issues] [Reporting] [Hand over] [VB Scripts] [JMeter tips] [JMeter RAW] [Dynatrace] [Documents] [FAQ] [Legal]

In the Cartesian Elements Ltd group of companies