It's been a long time since I've benchmarked a RDBMS, but it's always the same story: Benchmarks are rarely done correctly. People think a benchmark as the defi

It's been a long time since I've benchmarked a RDBMS, but it's always the same story: Benchmarks are rarely done correctly. People think a benchmark as the definitive way to prove their point, so they hastily and sloppily design a simplistic experiment to measure their desired quantities and then cite it all over the place as The Truth. For almost 8 years I've had to constantly scrutinize people's benchmarks (worked in networking and distributed systems research, and measurements are crucial in those contexts), point out their errors, and make them think about the details and "hidden" factors that affect their experiments. I haven't met more than a handful of people that could get a benchmark right on their own, and none (including myself) that get it right the first time. It's good that there are people that could point our errors out.