5 ms·

Well, the author cares enough about RethinkDB to test it, even if he's a mongodb fan, even if his first benchmark was wrong, he was right to publish it: you all

Well, the author cares enough about RethinkDB to test it, even if he's a mongodb fan, even if his first benchmark was wrong, he was right to publish it: you all helped him when you pinpointed the problems in his tests... Thanks you for that. I don't see any marketing here, just the "do your own benchmark" best practice, and the "share with community" best practice... Does it make it a perfect benchmark? No, but at least he tried... and the author has corrected the discrepancies since then. Now imagine the benchmark was against [your favorite DB here] with even stronger results against RethinkDB. Notice how the most upvoted comment is joking about MongoDB. The second one is a pro-mysql comment. What's the point? Would it have been a better benchmark if it read "mysql is 10x faster than RethinkDB?" or "MongoDB is even slower than RethinkDB"?

sarciszewski 11y ago
I have to say, it's rare for me to agree with a HN comment as much as I do to this one. Well said!
sklivvz1971 11y ago
I suppose the author of the test forgot the "only test realistic scenarios" best practice.
- weddpros 11y ago
  We know nothing about the author's use case... Maybe he wanted to know where each DB shines compared to each other, to see if some workloads are better suited to one or the other. Of course, benchmarks "should" include concurrent reads/updates/writes/deletes because it can make a huge difference depending on the DB's implementation. Of course, the author "should" also have tested sharding / durability / resistance to partition / resource consumption in his tests... Maybe he didn't have the resources to test properly. I also do quick&dirty benchmarks like these, mostly because exhaustive benchmarks cost so much more (time, money, expertise)...
timmaxw 11y ago
> the author has corrected the discrepancies since then As of the time I posted this comment, the blog post still seems to be comparing indexed MongoDB operations against non-indexed RethinkDB operations. Under those conditions I'd expect RethinkDB to be at least 1000x slower than MongoDB. The fact that he's finding that RethinkDB is only 3x slower than MongoDB makes me think that there are still other major problems with this benchmark. > No, but at least he tried... It's true that the author tried; but that doesn't change the fact that people are going to read this blog post and assume that the numbers are at least approximately correct. As a RethinkDB employee, it really frustrates me to see RethinkDB being judged according to benchmarks that are conducted so carelessly that they are essentially random. I think this is the fourth time in the past year that I've seen a third party try to benchmark RethinkDB and get something wrong. Maybe we need to start a "best practice" of checking in with the maintainers of a project before publishing benchmark results about the project.
- threeseed 11y ago
  Not sure why you are frustrated it's just a blog post by someone who was inexperienced with your product. At least he owned up to the mistakes and was willing to fix it. It's an opportunity for you to work with the guy to show him how to do it properly and write a blog post of your own. I would say that you probably should look at your API because I've never used a database that required me to explicitly define which indexes I want to use for a read. But I've never used RethinkDB so maybe there is a legitimate reason.
  - andrewflnr 11y ago
    GP is frustrated because it's not just the one blog post. Even if the author is willing to correct it, the original bad data will tend to get more exposure, because most people won't check in for corrections (unless they see a post like OP). Then, there'll be another crappy benchmark next month or next week. If I thought my product was being judged this way, it would drive me frothing mad.
- fche 11y ago
  @threeseed > [...] I've never used a database that required me to explicitly define which indexes I want to use for a read. [...] In a way, it's traditional (IBM IMS/DB, 1960s).
DannoHung 11y ago
> "do your own benchmark" best practice Is this a best practice? It seems like we've been delivered evidence that it is really hard to do good benchmarks unless you're already intimately familiar with what you're testing, which says something about how hard it is to make a good choice. I don't know about other industries, but this sort of result is what stuff like the STAC M3 Benchmark suite was designed for: Typical usecases that experts can implement so you can get realistic performance comparisons.
copsarebastards 11y ago
So basically you're saying that because he cares and because he tried, and because the comments were dumb, nobody should criticize him? Do you really not care about getting accurate results? Running benchmarks is an engineering practice. If you failed to get meaningful results, you failed. Yes, he cares, yes, he tried, yes, the comments are dumb, but he still failed. Sure, I'll give the guy kudos for trying, but I'm not going to pretend he didn't fail. As far as I'm concerned, telling someone they failed is a favor, because now they can change their methodology, try again, and maybe succeed. It's part of the process of achieving meaningful results. The entire point of what he's doing is to achieve meaningful results, not to get a participation medal. Your response reminds me of this: https://www.youtube.com/watch?v=gSjLiQxEZlM https://www.youtube.com/watch?v=gSjLiQxEZlM
- JDDunn9 11y ago
  You must be fun at parties...
  - copsarebastards 11y ago
    I am, because I'm there to have fun. Looking at DB benchmarks can be fun, but there are other reasons I do it.
- steego 11y ago
  > So basically you're saying that because he cares and because he tried, and because the comments were dumb, nobody should criticize him? I don't think anybody is saying that.
  - copsarebastards 11y ago
    Please, enlighten me on what you think weddpros is saying.
    - steego 11y ago
      Weddpros was making the point that criticism worked. The poster tried (An important first step), failed, and critical people pointed out his errors. Weddpros points out the poster corrected those errors because he received criticism. Weddpros is clearly a fan of critical feedback.

sarciszewski 11y ago

I have to say, it's rare for me to agree with a HN comment as much as I do to this one. Well said!

sklivvz1971 11y ago

I suppose the author of the test forgot the "only test realistic scenarios" best practice.

weddpros 11y ago
We know nothing about the author's use case... Maybe he wanted to know where each DB shines compared to each other, to see if some workloads are better suited to one or the other. Of course, benchmarks "should" include concurrent reads/updates/writes/deletes because it can make a huge difference depending on the DB's implementation. Of course, the author "should" also have tested sharding / durability / resistance to partition / resource consumption in his tests... Maybe he didn't have the resources to test properly. I also do quick&dirty benchmarks like these, mostly because exhaustive benchmarks cost so much more (time, money, expertise)...

timmaxw 11y ago

> the author has corrected the discrepancies since then As of the time I posted this comment, the blog post still seems to be comparing indexed MongoDB operations against non-indexed RethinkDB operations. Under those conditions I'd expect RethinkDB to be at least 1000x slower than MongoDB. The fact that he's finding that RethinkDB is only 3x slower than MongoDB makes me think that there are still other major problems with this benchmark. > No, but at least he tried... It's true that the author tried; but that doesn't change the fact that people are going to read this blog post and assume that the numbers are at least approximately correct. As a RethinkDB employee, it really frustrates me to see RethinkDB being judged according to benchmarks that are conducted so carelessly that they are essentially random. I think this is the fourth time in the past year that I've seen a third party try to benchmark RethinkDB and get something wrong. Maybe we need to start a "best practice" of checking in with the maintainers of a project before publishing benchmark results about the project.

threeseed 11y ago
Not sure why you are frustrated it's just a blog post by someone who was inexperienced with your product. At least he owned up to the mistakes and was willing to fix it. It's an opportunity for you to work with the guy to show him how to do it properly and write a blog post of your own. I would say that you probably should look at your API because I've never used a database that required me to explicitly define which indexes I want to use for a read. But I've never used RethinkDB so maybe there is a legitimate reason.
- andrewflnr 11y ago
  GP is frustrated because it's not just the one blog post. Even if the author is willing to correct it, the original bad data will tend to get more exposure, because most people won't check in for corrections (unless they see a post like OP). Then, there'll be another crappy benchmark next month or next week. If I thought my product was being judged this way, it would drive me frothing mad.
fche 11y ago
@threeseed > [...] I've never used a database that required me to explicitly define which indexes I want to use for a read. [...] In a way, it's traditional (IBM IMS/DB, 1960s).

DannoHung 11y ago

> "do your own benchmark" best practice Is this a best practice? It seems like we've been delivered evidence that it is really hard to do good benchmarks unless you're already intimately familiar with what you're testing, which says something about how hard it is to make a good choice. I don't know about other industries, but this sort of result is what stuff like the STAC M3 Benchmark suite was designed for: Typical usecases that experts can implement so you can get realistic performance comparisons.

copsarebastards 11y ago

So basically you're saying that because he cares and because he tried, and because the comments were dumb, nobody should criticize him? Do you really not care about getting accurate results? Running benchmarks is an engineering practice. If you failed to get meaningful results, you failed. Yes, he cares, yes, he tried, yes, the comments are dumb, but he still failed. Sure, I'll give the guy kudos for trying, but I'm not going to pretend he didn't fail. As far as I'm concerned, telling someone they failed is a favor, because now they can change their methodology, try again, and maybe succeed. It's part of the process of achieving meaningful results. The entire point of what he's doing is to achieve meaningful results, not to get a participation medal. Your response reminds me of this: https://www.youtube.com/watch?v=gSjLiQxEZlM https://www.youtube.com/watch?v=gSjLiQxEZlM

JDDunn9 11y ago
You must be fun at parties...
- copsarebastards 11y ago
  I am, because I'm there to have fun. Looking at DB benchmarks can be fun, but there are other reasons I do it.
steego 11y ago
> So basically you're saying that because he cares and because he tried, and because the comments were dumb, nobody should criticize him? I don't think anybody is saying that.
- copsarebastards 11y ago
  Please, enlighten me on what you think weddpros is saying.
  - steego 11y ago
    Weddpros was making the point that criticism worked. The poster tried (An important first step), failed, and critical people pointed out his errors. Weddpros points out the poster corrected those errors because he received criticism. Weddpros is clearly a fan of critical feedback.