4 ms·

In your 2015 article, you criticized that the ArangoDB team restarted the instances after each test run. In this 2018 edition, they don't do this anymore.

maxdemarzi 9y ago
You want to trust a vendor that used to restart the db between queries for their benchmarks?
- virmundi 9y ago
  Yes, because they learned. Arango has honest people that are open to criticism.
  - tinymollusk 9y ago
    But the larger issue -- that they have direct financial incentive to unconsciously bias the results -- still exists. This isn't anything against Arango's team, I'm sure they're lovely people, but even physics experimental results have been shown to be biased when the experimenter knew the testing hypothesis. "I think I've been in the top 5% of my age cohort all my life in understanding the power of incentives, and all my life I've underestimated it. Never a year passes that I don't get some surprise that pushes my limit a little farther." -- Charlie Munger
    - JackFr 9y ago
      Or Feynman: "The first principle is that you must not fool yourself and you are the easiest person to fool."
    - jrs95 9y ago
      True, but so does everybody else. You shouldn't trust vendor benchmarks, but that doesn't mean you can't read them and use them as sort of a heuristic to help you come to your own conclusions. It might not be particularly scientific, but it's probably more than accurate enough for 99% of people.
      - tinymollusk 9y ago
        True, but that is because every major database is good enough for 99% of people even when run on the cheapest linode available ;).
    - ifcologne 9y ago
      Performance tests - especially those of databases - are a very complex and resource-consuming venture. And because each use case is different, the benchmark published somewhere does not fit your specific problem and the available environment/budget. Unfortunately, there is no independent organization that believes in this and defines scenarios that are tested in different environments.
      - tinymollusk 9y ago
        Very good points. Seems like this leaves us with "trust benchmarks that probably aren't generalizable to your use-case" or "run your own expensive benchmarks before you have the scale to have the data to simulate the scale". What's the best strategy here? I've defaulted to MySQL, pg, or sqlite, based on my preference of the moment. I haven't had enough good/bad outcomes to form a stronger opinion, but I'm now realizing I've never really thought this decision through (despite building a few data systems that I'm pretty proud of).
      - janemanos 9y ago
        We just published an Update to the Benchmark, please find it here: https://news.ycombinator.com/item?id=16473117 https://news.ycombinator.com/item?id=16473117
  - virmundi 9y ago
    I trust but verify. When dealing with Arango I’m Not openly hostile to their claims. I’ve made several libraries for it even though I don’t use it. Contrast this with Mongo. Their web scale carpet bombing campaign makes me not use them at all. I immediately dismiss them. That is how I found Arango during my booklet writing days.

maxdemarzi 9y ago

You want to trust a vendor that used to restart the db between queries for their benchmarks?

virmundi 9y ago
Yes, because they learned. Arango has honest people that are open to criticism.
- tinymollusk 9y ago
  But the larger issue -- that they have direct financial incentive to unconsciously bias the results -- still exists. This isn't anything against Arango's team, I'm sure they're lovely people, but even physics experimental results have been shown to be biased when the experimenter knew the testing hypothesis. "I think I've been in the top 5% of my age cohort all my life in understanding the power of incentives, and all my life I've underestimated it. Never a year passes that I don't get some surprise that pushes my limit a little farther." -- Charlie Munger
  - JackFr 9y ago
    Or Feynman: "The first principle is that you must not fool yourself and you are the easiest person to fool."
  - jrs95 9y ago
    True, but so does everybody else. You shouldn't trust vendor benchmarks, but that doesn't mean you can't read them and use them as sort of a heuristic to help you come to your own conclusions. It might not be particularly scientific, but it's probably more than accurate enough for 99% of people.
    - tinymollusk 9y ago
      True, but that is because every major database is good enough for 99% of people even when run on the cheapest linode available ;).
  - ifcologne 9y ago
    Performance tests - especially those of databases - are a very complex and resource-consuming venture. And because each use case is different, the benchmark published somewhere does not fit your specific problem and the available environment/budget. Unfortunately, there is no independent organization that believes in this and defines scenarios that are tested in different environments.
    - tinymollusk 9y ago
      Very good points. Seems like this leaves us with "trust benchmarks that probably aren't generalizable to your use-case" or "run your own expensive benchmarks before you have the scale to have the data to simulate the scale". What's the best strategy here? I've defaulted to MySQL, pg, or sqlite, based on my preference of the moment. I haven't had enough good/bad outcomes to form a stronger opinion, but I'm now realizing I've never really thought this decision through (despite building a few data systems that I'm pretty proud of).
    - janemanos 9y ago
      We just published an Update to the Benchmark, please find it here: https://news.ycombinator.com/item?id=16473117 https://news.ycombinator.com/item?id=16473117
- virmundi 9y ago
  I trust but verify. When dealing with Arango I’m Not openly hostile to their claims. I’ve made several libraries for it even though I don’t use it. Contrast this with Mongo. Their web scale carpet bombing campaign makes me not use them at all. I immediately dismiss them. That is how I found Arango during my booklet writing days.