Also measure many times, and make sure any difference you see isn't accounted for by variation in the test system (CPU frequency scaling due to temperature, you

Also measure many times, and make sure any difference you see isn't accounted for by variation in the test system (CPU frequency scaling due to temperature, your test being run partly (or entirely) on low-power cores if your CPU has such a distinction, network contention if you aren't working locally or over a link dedicated to the tests, etc.). Also where IO is at all involved remember to test both cold and warm starts, and make sure that if your new version seems faster in the results that it actually is rather than a previous run having “warmed up” something. Make sure any prep you do before each measurement doesn't inadvertently change the results – keep prep and actual the tests separate (and cool down between if trying to analyse cold-run performance). On this latter point I once had a heck of a time convincing someone that his new index hadn't sped up a SQL query by an order of magnitude – he was correctly running from cold each time but was adding the index and running the query immediately (without dropping cached pages between), the affected table was small enough to fit in memory and adding the index caused a full scan so when the actual test was done the system was no longer cold.