The results from last performance test outperformed even optimistic expectations?
Before publishing those test results we recommend to check, if you did run into one of the following test traps:

Trap 1: Unrealistic Cache Hits

in

• Database Cache
• File System Cache
• SAN – Cache

when repeating test runs with same test data but different testing parameters - a typical scenario during extensive performance-, load- and stress testing.

How to identify

Cache hits in the file system cache and in the SAN-cache can be identified by the disk latency reported in the database performance statistics.
Disk latency of less then 5 milliseconds for reading one 8 KB or 16 KB database block are “too fast” for a real spindle-hit.
5 milliseconds latency are excellent values, 6-8 milliseconds are good.
For the case that you do not expect such (high) cache hits in file system cache and SAN cache in realistic production scenario, this fact needs to be reported.

How to avoid

Database Cache: Before each tests flush the database cache; if your database does not provide such a command, you need to restart the database.
File System Cache: Reboot the server.
SAN-Cache: for the case that you do not expect cache hits on the SAN storage in real production cycle, you need to include in your performance test results not only the measured raw data but also an estimation of slower but realistic results considering real production scenarios. You might also discuss with your SAN-vendor this topic, e.g. if caching can be reduced during your test.

Trap 2: “sorted” test data

Pre-requisite for performance tests is the creation of sufficient volume of test data – in optimal case the full volume of customers.

For that purpose scripts are used to create artificial customers, e.g.
customer_id = 1, name = “Customer1”
customer_id = 2, name = “Customer2”.

Assuming an average record length of 200 bytes, a 8 KB database block will contain data of about 40 customers.

If in real live those customers create randomly transactions, and due to the large size of your database table(s) containing customer data only a small percentage of those table(s) is in the database cache, then this will result in physical reads; and in most cases only 1 of the 40 customers in the database block read from disk will be accessed.

However, if your transaction test data for your performance test contain the transactions ordered by customer id, then during the test all 40 customer records will be accessed in the short time this database block remains in the database cache.

How to avoid

Create test data with sufficient randomness, and ensure the same randomness during processing those test data.