I am wondering what execution time is still ok tested with ab.exe considering that a script will be run on each pageview. And how can you know if execution time is because a script and query were cached.
For example if my I test in ab.exe script takes 0.156 seconds. If I try again it showes 0.047s. If I make query in phpmyadmin, it showes execution time 0.036 (probably query was already cached).
I am really confused how to test it and how to know what was cached.
It’s more than just mysql’s query cache(you can disable that). mysql can also give you a lot of statistics on how many query cache hits/misses etc…see the mysql docs.
But, the real problem is the operating system, and disks also cache files(or parts of them).
This makes it real hard to benchmark a specific thing in isolation because you really can’t reproduce how it will be working on a live site. You need to get those disks seeking like hell so the caches can’t trivialize it.
A drawback of apachebench is it’s not easy to test different urls. But you want to, because you need to get your disks seeking to all the different places they’re going to be when in production, otherwise caching will give you a false impression of performance.
Siege looks promising for being able to get different urls in there. This way you can send different get/post etc… combinations.
But, disabling mysql’s query cache, and making sure to issue variations on the queries that would need to read from many different parts of the disk, might be a simpler but adequate approach.