1. Final results analysis aimed at CI build pass/fail
The following metrics can be set up against any number of specific transactions in your model:
a. 95th percentile calculated and tested against given limit. Build fails if this criteria is not met for this transaction
b. Requests percentage pass/fail tested against given limit e.g. 90% passes for transaction 1, or the build fails
c. Pass TPS levels tested against limits. e.g. transaction 2 must average at least 120 transactions passing per second for the build to pass
d. Fail TPS levels tested against limits. e.g. transaction 3 must average UNDER 10 transactions per second fails for the build to pass
c. General percentile check. For a particular transaction, check the passes or fails or both at a specified percentile e.g. 85th percentile of passes must come in under the set limit. Build will fail if this criteria is not met. This uses different data files than 1a. above. see here for details.
2. Runtime data analysis
The following metrics can be reported for any number of specific transactions in your model. These are reported at specified intervals (e.g. 15 seconds) and are calculated over a short (configurable) time span so constitute local averages rather than overall numbers as per the final results above. This is useful for monitoring test progress.
a. pass count
b. fail count
c. pass rate
d. fail rate
e. 90th percentile (this could be changed within the script if required)
and elapsed scenario time is also reported here.