I want to run tests straight from raw log files from our app servers. First steps can be seen here.
The main reason for this approach is to keep test data up-to-date and keep request ratios relevant to recent site activity.
This is mainly for regression testing in our stage environments. I can pull access logs from production and do very little work on them before using them to drive our latest round of testing.
And rather than just replaying logs, this approach provides timings for various groups of calls, allowing us to hone in on any potential issues with the code at an early stage.
So this is a general performance model BUT bespoke to our project needs.
This page will go through the model's approach and you can download the source code above.
We start with a (cleaned up) application access log file (behind the cache) - typically something like 900,000 rows per server per day:
210.155.zz.uu - - [26/Nov/2014:06:26:44 +0000] "GET /sta/130x73/title4/79866028-e11e-4014-9702-9a2cca39d93a_625x352.jpg HTTP/1.1" 200 4593 "http://www.<server>.com/programmes/title2/videos/all/s4-ep4-great-fulford" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.1.25 (KHTML, like Gecko) Version/8.0 Safari/600.1.25" - 0.086
2.20.xx.yy - - [26/Nov/2014:06:26:44 +0000] "GET /proj/160x90/videos/b1f36bd5-5acd-4d76-be9f-ce921bf34b11.jpg HTTP/1.1" 200 3859 "http://www.<server>.com/programmes/title1" "Mozilla/5.0 (iPad; CPU OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53" - 0.039
209.148.vv.ww - - [26/Nov/2014:06:26:44 +0000] "GET /programmes/service/brand/title3.xml HTTP/1.1" 200 1984 "http://www.<server>.com/programmes/title3/on-demand/57726-001" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36" - 0.000
We do one thing to this file to drastically cut down our data transfer requirements: we pull out the main field we really need for our jmeter script: field 7:
This gives us something like this (900,000 rows):
Then we come to the jmeter script: