Iím a LoadRunner man. Iíve used it for more than 10 years and it is excellent. A first class product. Click click click and youíve got a script, click click click and youíve got it parameterised (wait till youíve used the alternatives!). A bit more work and youíve got data handling down to a tee. And Analysis tools to die for.
The only problem with this is you get what you pay for and LoadRunner is expensive, often outside the budgets of even some large organizations.
So recently Iíve been using Facilita Forecast. And the consultancy group Intechnica can supply you with a very neat tool called Traffic Spike which gives you easy control on cloud based load injectors that makes this tool infinitely scalable.
Facilita Forecast is a great tool. Itís a C++ based product so uses native code so is highly efficient and you can scale it up as with LoadRunner on any decent hardware. But it is basic compared to LoadRunner, several steps down the tree. You need to be coding everything which is fine but of course time consuming. The analysis is limited. I recommend becoming a Splunk expert as well to help on the analysis side - the free version can be used successfully for small projects.
But even Forecast has a cost and when it comes to CI performance testing for example, the cost of controllers can become an issue. This is because CI projects necessarily tie up controllers on the continuous cycle, with each project ideally having itís own resources.
So this is where JMeter comes in. I am also a JMeter expert! (woo hoo I hear you say). Well, I have been using JMeter for CI for a number of years now. Itís good at low scale testing. The problem is the Java virtual machine it sits in. It just isnít scalable. Yes you can tweak it but basically you end up testing Java rather than your application. And yes you can use a master slave setup but I didnít even go down this route with all the difficulties Iíve heard of.
But now, we can come bang up to date (as of February 2013) and start using the cloud. AWS is actually pretty straight forward to use and is quite cheap for our purposes, particularly as we can bring up resources and tear them down quickly as and when we need them.
I started with Oliverís excellent script for cloud based JMeter testing. But I need a few more bells and whistles. In particular I need more results information and I need some CI pass fail requirements. Read on...