4 ms·

But the larger issue -- that they have direct financial incentive to unconsciously bias the results -- still exists. This isn't anything against Arango's team,

But the larger issue -- that they have direct financial incentive to unconsciously bias the results -- still exists. This isn't anything against Arango's team, I'm sure they're lovely people, but even physics experimental results have been shown to be biased when the experimenter knew the testing hypothesis. "I think I've been in the top 5% of my age cohort all my life in understanding the power of incentives, and all my life I've underestimated it. Never a year passes that I don't get some surprise that pushes my limit a little farther." -- Charlie Munger

JackFr 9y ago
Or Feynman: "The first principle is that you must not fool yourself and you are the easiest person to fool."
jrs95 9y ago
True, but so does everybody else. You shouldn't trust vendor benchmarks, but that doesn't mean you can't read them and use them as sort of a heuristic to help you come to your own conclusions. It might not be particularly scientific, but it's probably more than accurate enough for 99% of people.
- tinymollusk 9y ago
  True, but that is because every major database is good enough for 99% of people even when run on the cheapest linode available ;).
ifcologne 9y ago
Performance tests - especially those of databases - are a very complex and resource-consuming venture. And because each use case is different, the benchmark published somewhere does not fit your specific problem and the available environment/budget. Unfortunately, there is no independent organization that believes in this and defines scenarios that are tested in different environments.
- tinymollusk 9y ago
  Very good points. Seems like this leaves us with "trust benchmarks that probably aren't generalizable to your use-case" or "run your own expensive benchmarks before you have the scale to have the data to simulate the scale". What's the best strategy here? I've defaulted to MySQL, pg, or sqlite, based on my preference of the moment. I haven't had enough good/bad outcomes to form a stronger opinion, but I'm now realizing I've never really thought this decision through (despite building a few data systems that I'm pretty proud of).
- janemanos 9y ago
  We just published an Update to the Benchmark, please find it here: https://news.ycombinator.com/item?id=16473117 https://news.ycombinator.com/item?id=16473117

JackFr 9y ago

Or Feynman: "The first principle is that you must not fool yourself and you are the easiest person to fool."

jrs95 9y ago

True, but so does everybody else. You shouldn't trust vendor benchmarks, but that doesn't mean you can't read them and use them as sort of a heuristic to help you come to your own conclusions. It might not be particularly scientific, but it's probably more than accurate enough for 99% of people.

tinymollusk 9y ago
True, but that is because every major database is good enough for 99% of people even when run on the cheapest linode available ;).

ifcologne 9y ago

Performance tests - especially those of databases - are a very complex and resource-consuming venture. And because each use case is different, the benchmark published somewhere does not fit your specific problem and the available environment/budget. Unfortunately, there is no independent organization that believes in this and defines scenarios that are tested in different environments.

tinymollusk 9y ago
Very good points. Seems like this leaves us with "trust benchmarks that probably aren't generalizable to your use-case" or "run your own expensive benchmarks before you have the scale to have the data to simulate the scale". What's the best strategy here? I've defaulted to MySQL, pg, or sqlite, based on my preference of the moment. I haven't had enough good/bad outcomes to form a stronger opinion, but I'm now realizing I've never really thought this decision through (despite building a few data systems that I'm pretty proud of).
janemanos 9y ago
We just published an Update to the Benchmark, please find it here: https://news.ycombinator.com/item?id=16473117 https://news.ycombinator.com/item?id=16473117