3 ms·

The benchmark you’re reading is probably wrong

37 points by coffeemug 16y ago

antirez 16y ago
Another common flaw in broken benchmarks: use a single client instead of many simultaneous clients. In this configuration you are mostly measuring the RTT plus the client lib implementation latency. Another one is testing on-disk DBs with datasets that fit in RAM: the whole DB data will be cached in OS buffers and you don't get the picture of how the DB will perform with big data. Benchmarking is one of the most complex tasks, but for some reason benchmarking is usually not done by the most expert guys in the field.
mfukar 16y ago
It's been a long time since I've benchmarked a RDBMS, but it's always the same story: Benchmarks are rarely done correctly. People think a benchmark as the definitive way to prove their point, so they hastily and sloppily design a simplistic experiment to measure their desired quantities and then cite it all over the place as The Truth. For almost 8 years I've had to constantly scrutinize people's benchmarks (worked in networking and distributed systems research, and measurements are crucial in those contexts), point out their errors, and make them think about the details and "hidden" factors that affect their experiments. I haven't met more than a handful of people that could get a benchmark right on their own, and none (including myself) that get it right the first time. It's good that there are people that could point our errors out.
nuxi 16y ago
Ironically, this blog wasn't benchmarked either: 504 Gateway Time-out nginx/0.6.35 The main page works though.
- bradleyland 16y ago
  Given that it's a 504 error, it looks like the app server died or is out of threads/processes. It's hard to take technical advice from someone who doesn't do simple things like enable caching in their CMS. Chances are nginx would still be serving the content if it were a cached resource.
  - mahmud 16y ago
    You would be wrong to dismiss the blog post and its author.
    - bradleyland 16y ago
      I cannot dismiss what I cannot access.
      - mahmud 16y ago
        The front page still works, go there and look around. FWIW, this fancy ginger cat is their CTO: http://www.defmacro.org/ http://www.defmacro.org/ And I have it on good authority that he knows a thing or two about tweaking a "CMS" ;-)

37 points by coffeemug 16y ago

Another common flaw in broken benchmarks: use a single client instead of many simultaneous clients. In this configuration you are mostly measuring the RTT plus the client lib implementation latency. Another one is testing on-disk DBs with datasets that fit in RAM: the whole DB data will be cached in OS buffers and you don't get the picture of how the DB will perform with big data. Benchmarking is one of the most complex tasks, but for some reason benchmarking is usually not done by the most expert guys in the field.

mfukar 16y ago

It's been a long time since I've benchmarked a RDBMS, but it's always the same story: Benchmarks are rarely done correctly. People think a benchmark as the definitive way to prove their point, so they hastily and sloppily design a simplistic experiment to measure their desired quantities and then cite it all over the place as The Truth. For almost 8 years I've had to constantly scrutinize people's benchmarks (worked in networking and distributed systems research, and measurements are crucial in those contexts), point out their errors, and make them think about the details and "hidden" factors that affect their experiments. I haven't met more than a handful of people that could get a benchmark right on their own, and none (including myself) that get it right the first time. It's good that there are people that could point our errors out.

nuxi 16y ago

Ironically, this blog wasn't benchmarked either: 504 Gateway Time-out nginx/0.6.35 The main page works though.

bradleyland 16y ago
Given that it's a 504 error, it looks like the app server died or is out of threads/processes. It's hard to take technical advice from someone who doesn't do simple things like enable caching in their CMS. Chances are nginx would still be serving the content if it were a cached resource.
- mahmud 16y ago
  You would be wrong to dismiss the blog post and its author.
  - bradleyland 16y ago
    I cannot dismiss what I cannot access.
    - mahmud 16y ago
      The front page still works, go there and look around. FWIW, this fancy ginger cat is their CTO: http://www.defmacro.org/ http://www.defmacro.org/ And I have it on good authority that he knows a thing or two about tweaking a "CMS" ;-)