LR Rules OK

Why LoadRunner is worth the money

This page is of course from a professional load testerís perspective, rather than the project managerís view, who may have to pay for it!

I have used LoadRunner for more than 10 years but recently I have been at a location where LoadRunner just wasnít in the budget. So we used TestPlant Eggplant (formerly Facilita Forecast) and JMeter. These tools are ok, in fact you could say they are better value for money than loadRunner, but this is only because they are cheap or free. Mind you, I donít want to put these two tools down here, not at all. They are valuable contributions to the space and I have been working commercially for 2 1/2 years with only these tools and we have kept commercial scale sites up and running.

However, if you can run to the expense of HP LoadRunner, then it really is well worth it. And hereís why:

NOTE: Iíll compile this page as issues present themselves along the lines of a blog. I am currently replacing my eggplant and jmeter workflows with LoadRunner ones. This will be a progression over the next few months, particularly with the jmeter CI work.

1. LoadRunner now supports CI with Jenkins

2. eggPlant often reports misleading numbers

3. eggPlant doesnít fully support long test runs

4. LoadRunner logging is great

5. LR runs for days...

6. LR analysis compares runs

7. LR analysis has some great facilities, such as time to first byte breakdown

8. LR allows users to be ramped up/down on the fly without stopping the test

9. LoadRunner has complete control over graphs, with comments and 3D and many other options available

 

1. LoadRunner now supports CI with Jenkins

This is the major reason why we have been able to reconsider LoadRunner at work. We need CI and jmeter does have severe limitations, especially with scaling up. There is always a question with jmeter around JVM memory management and how GC can affect results and with CPU contention. LoadRunner doesnít have this issue of course (native code) and now that LoadRunner fully supports CI with Jenkins, this becomes a real contender for our attentions. See LR Slave and LR CI Graphs for more details and now LoadRunner CI for a fully working solution..

2. eggPlant often reports misleading numbers

In Eggplant you often get misleading numbers reported, for example in todayís test run Iíve got two rows for the same metric, each reporting different TPS rates and response times (!!!). Luckily in this case I can reasonably guess which one is correct as I know the system but often it is a closer call than this:

image00114

3. eggPlant doesnít fully support long test runs

I recently ran a 48 hour test in eggplant and LoadRunner. LoadRunner is currently churning through the results:

image00211

And in the meantime gives us a summary graph:

image00308

Eggplant canít handle this length of test and I have to turn off logging and rely on screenshots of the runtime metrics, which are not as accurate (see above!):

image00509

4. LoadRunner logging is great

In LR, we can view the logs during runtime and the log window gives us the path of the log, which is a simple text file. In eggplant, the software usually gets stuck trying to see the logs and they are not simple text files so canít be opened easily. Eggplant just loses valuable information and this makes it hard  to support the devs with issues under load.

Here is an example where I have 16 failures out of 3,530 transactions. In eggplant I simply couldnít find any information  about those failures. In LoadRunner I have the logs, can open them in my favourite text editor and go straight to the issue:

image00611
image00710

So, within 5 minutes a discussion with the dev found the issue on the site and it turns out its an edge case and the site behaves correctly. This led to  a change in the load test script:

 

               //front end doesn't allow users to send down empty string (like google)

               if(strcmp(lr_eval_string("{word}"),"") == 0)

                               return 0;

 

And issue is resolved.

5. LR runs for days...

LR is designed to run for days. This seems so obvious but when you set a number of hours in eggplant that is more than 24, it does run for that period (3 days say) but the graphs and analysis canít handle the concept of days and hours just go back to modulus(24). Here is the LR scenario timing box:

image00805

6. LR analysis compares runs

LR allows you to compare runs to each other. You can load up several sets of tests, say after dev changes, and compare different graphs between runs. A simple example here is response times with dev changes during a typical tuning investigation. eggplant and jmeter simply donít have this facility.

image00904

7. LR analysis has some great facilities, such as time to first byte breakdown

Time to first byte breakdown shows the network latency versus server time. In this case we had some configuration issues we were trying to track down. Again, this sort of detail just isnít available in the other test tool suites. Our dev explained this as follows:

    Yeah. Every HTTP response comes back in TCP packets and the tool is measuring the difference between the time taken to receive the first packet vs the last packet. The difference between these two is the network latency. The server time is the time it takes for the server to respond with the very first packet of data. In this case I would expect the majority of the time to be server side  based on the programme core-service reaching its cache limit and requests therefore flooding back to PML.

Two screenshots here to more clearly show the split between network time and server time for first byte:

image01004
image011

8. LR allows users to be ramped up/down on the fly without stopping the test

In LoadRunner you can start/stop any users you wish during the test run. This is very useful when working with dev teams, perhaps monitoring progress, trying to achieve particular back end requirements. You cannot do this in eggPlant or jmeter. And LoadRunner gives quite fine control, for instance you can initialise users separately from starting them running which can be very useful for certain scenarios, perhaps with heavy data requirements on user startup:

image012
image013

9. LoadRunner has complete control over graphs, with comments and 3D and many other options available

image015
image016
image01402

[Home] [About (CV)] [Contact Us] [JMeter Cloud] [webPageTest] [_64 images] [asset moniitor] [Linux Monitor] [Splunk ETL] [Splunk API] [AWS bash] [LR Rules OK] [LR Slave] [LR CI Graphs] [LoadRunner CI] [LR CI Variables] [LR Bamboo] [LR Methods] [LR CI BASH] [Bash methods] [Jenkins V2] [Streaming vid] [How fast] [Finding Issues] [Reporting] [Hand over] [VB Scripts] [JMeter tips] [JMeter RAW] [Dynatrace] [Documents] [FAQ] [Legal]

In the Cartesian Elements Ltd group of companies