It's more than just mysql's query cache(you can disable that). mysql can also give you a lot of statistics on how many query cache hits/misses etc..see the mysql docs.
But, the real problem is the operating system, and disks also cache files(or parts of them).
This makes it real hard to benchmark a specific thing in isolation because you really can't reproduce how it will be working on a live site. You need to get those disks seeking like hell so the caches can't trivialize it.
A drawback of apachebench is it's not easy to test different urls. But you want to, because you need to get your disks seeking to all the different places they're going to be when in production, otherwise caching will give you a false impression of performance.
Siege looks promising for being able to get different urls in there. This way you can send different get/post etc... combinations.
But, disabling mysql's query cache, and making sure to issue variations on the queries that would need to read from many different parts of the disk, might be a simpler but adequate approach.