11 ms·

“The benchmark numbers are completely wrong for both databases”

165 points by neumino 11y ago

resonation 11y ago
...forgot to call run().
- threeseed 11y ago
  It was rerun with that call: https://www.amon.cx/blog/rethinkdb-reviewed-by-a-mongo-fan/ https://www.amon.cx/blog/rethinkdb-reviewed-by-a-mongo-fan/ Still we don't know the hardware, what versions were installed, configuration changes etc. The whole thing is pretty meaningless.
jakozaur 11y ago
Benchmarking is hard and a lot of reports are bogus. However they are still very useful for a lot of developers. Benchmarking programming languages got better. E.g.: http://benchmarksgame.alioth.debian.org/ http://benchmarksgame.alioth.debian.org/ gives roughly idea about performance of programing languages. I wish something similar existed for databases. I think exact figures would be hard to get, but I believe there are many 2x 10x differences that we should be aware of.
- collyw 11y ago
  In my experience, the database is the best place to perform optimizations, not in the code. I guess a lot depends on what you are doing.
- amelius 11y ago
  > Benchmarking programming languages got better. I wish we got this kind of benchmark for the altjs solutions.
- boomlinde 11y ago
  > Benchmarking is hard and a lot of reports are bogus. However they are still very useful for a lot of developers. In this case we were presented with a benchmark setup that failed to perform the task it supposedly benchmarked. That's not hard to avoid, and it makes the benchmark completely useless and misleading. I don't think that this problem can be generalized in the sense you do here, since the problem with benchmarks usually isn't a complete failure to perform the task to be benchmarked, but things like finding a set of tests that give a fair representation of what you'd typically use the subjects for, or performing the tasks in idiomatic and optimal ways.
- amaranth 11y ago
  Don't a lot of those benchmarks end up only measuring how fast your language can call out to GMP to do the real work? And regex-dna ends up measuring your regex implementation which for a lot of them is again just going to be measuring how fast they can all call out to PCRE. They're neat and all and it is called the benchmarks game but I wish they'd remove the ones that end up getting gamed like that.
  - igouy 11y ago
    >> Don't a lot of those benchmarks… << No. >> …how fast they can all call out to PCRE. << No. Have you looked at the programs?
weavie 11y ago
> I expect to be able to click on a table and see the rows inside - like all the tools out there for Mongo, MySQL, PostgreSQL, etc. He does have a point there. It is slightly annoying having to type out a query when I just want to browse the data. (No real biggie though..)
- CHY872 11y ago
  It's not a good point, though. Writing a rudimentary tool of that type would take perhaps an hour.
  - weavie 11y ago
    Indeed, and typing out the query really doesn't take that long. It does affect the initial experience though. The admin panel looks so slick I just assumed I would be able to click on the table and it would jump to the data. When it didn't I was surprised and initially blamed myself and fired up the console to see if there were any errors showing. It just affects the polish of the panel. As I said though it is a minor point for an excellent product.
- neumino 11y ago
  There are other tools for this, like Chateau: https://github.com/neumino/chateau https://github.com/neumino/chateau
tinussky 11y ago
benchmark = marketing (more times than not)
- cpeterso 11y ago
  Hence "benchmarketing".
- threeseed 11y ago
  Everyone likes to see how their tool of choice performs against other tools. And it's an important (albeit just one part) of deciding which product to use.
  - tinussky 11y ago
    I always say do your own benchmarking for your own use case. The risk of a colored benchmark is quite high when benchmark is done by owner of product or by "fan" of product. With the exception of a well explained, clear benchmark that everyone can understand and reproduce easily.
    - davidw 11y ago
      At least with a public benchmark, people can point out flaws. With something rolled out internally, you're still likely to get flaws, but no one will point out that you misconfigured Postgres or set up Mongo the wrong way, or any of the other errors you are just as likely to make by doing it yourself. Perhaps, once you've winnowed the choices down to just a few, it might make more sense, but I think good public benchmarks can be a helpful thing for that selection process.
      - tinussky 11y ago
        I agree, public benchmarks do have their uses as i noted in my reply: "With the exception of a well explained, clear benchmark that everyone can understand and reproduce easily" An internal benchmark indeed requires knowledge on the subject, but most of the times people at the mailinglists are quite helpful when you explain what you are trying to do. Especially when you post a benchmark which is not in favorite of their product.
webtards 11y ago
Not sure what's worse here - people relying on third party benchmarks (hint: always do your own; see how a tool performs on your data, on your hardware, for your problem set), or the fanboy-ish panic when they are unsettled that a benchmark might make their chosen toy less shiny?
- mikecmpbll 11y ago
  The former :)
dantiberian 11y ago
I've been working with RethinkDB recently on some slightly unusual things and the Rethink team has been first class. They've got great support on IRC and GitHub and are open and friendly. I highly recommend them.
kalleboo 11y ago
From the linked benchmark: > In RethinDB, you have to create databases and tables manually and it will raise an exception if they already exist. Compared to MongoDB that could be an inconvenience for some(and me) - one of the things I find appealing in MongoDB is the fluid interaction with databases ... well at least now I don't feel so bad about having some old MySQL stuff still in production. MySQL already has too much "fluidity" in dealing with my data...
sklivvz1971 11y ago
This old comic piece seems to still apply: http://www.mongodb-is-web-scale.com/ http://www.mongodb-is-web-scale.com/
- weddpros 11y ago
  Then how on earth is it possible to build Parse.com with MongoDB? http://blog.parse.com/announcements/mongodb-rocksdb-parse/ http://blog.parse.com/announcements/mongodb-rocksdb-parse/
  - sklivvz1971 11y ago
    Maybe you didn't get the joke. The joke is not about MongoDB, but about MongoDB fanbois that care only about some very narrow definition of "performance". The wider message is that DBs are way more complex beasts that is meaningful to test this way. Obviously there are cases in which MongoDB is a great choice, but equally obviously tests like this should not be a reason for the choice.
    - weddpros 11y ago
      I do understand that second degree, but I don't think the author is a fanboy... At work, we're also Mongodb users. If tomorrow we try to benchmark against Cassandra, performance will probably be the selling point. I don't think it's absurd to compare mongodb and rethinkdb, they're very similar dbs. As for the benchmark, I agree it doesn't explore every facet of both databases and focus on performance... That's what benchmarks do. The intent of the author may have been to challenge his existing choice (mongodb) which is a good thing. The (corrected) results may lead to: expect no performance gain if we migrate to RethinkDB... What's wrong with that?
      - sklivvz1971 11y ago
        Well, OK. I am not calling the author a fanboy, and i'll agree they probably aren't. There are at least 3 things here which invalidate this post because they make the numbers uncomparable: 1. Is the hardware adequate to run the tests, or does it favor one database? 2. Are both databases tuned to the use case? 3. Running only one type of action at a time is meaningless. Contention between reads and writes is what normally drives performance stats.
JDDunn9 11y ago
Would be nice to see some official benchmarks from RethinkDB. As this illustrates, there aren't any good ones out there.
- insaneirish 11y ago
  > Would be nice to see some official benchmarks from RethinkDB. Of some artificial scenario that people will claim does not reflect [their] reality? > As this illustrates, there aren't any good ones out there. The only good benchmarks are the ones that you define to be representative of something you'll actually see in reality.
weddpros 11y ago
Well, the author cares enough about RethinkDB to test it, even if he's a mongodb fan, even if his first benchmark was wrong, he was right to publish it: you all helped him when you pinpointed the problems in his tests... Thanks you for that. I don't see any marketing here, just the "do your own benchmark" best practice, and the "share with community" best practice... Does it make it a perfect benchmark? No, but at least he tried... and the author has corrected the discrepancies since then. Now imagine the benchmark was against [your favorite DB here] with even stronger results against RethinkDB. Notice how the most upvoted comment is joking about MongoDB. The second one is a pro-mysql comment. What's the point? Would it have been a better benchmark if it read "mysql is 10x faster than RethinkDB?" or "MongoDB is even slower than RethinkDB"?
- sarciszewski 11y ago
  I have to say, it's rare for me to agree with a HN comment as much as I do to this one. Well said!
- sklivvz1971 11y ago
  I suppose the author of the test forgot the "only test realistic scenarios" best practice.
  - weddpros 11y ago
    We know nothing about the author's use case... Maybe he wanted to know where each DB shines compared to each other, to see if some workloads are better suited to one or the other. Of course, benchmarks "should" include concurrent reads/updates/writes/deletes because it can make a huge difference depending on the DB's implementation. Of course, the author "should" also have tested sharding / durability / resistance to partition / resource consumption in his tests... Maybe he didn't have the resources to test properly. I also do quick&dirty benchmarks like these, mostly because exhaustive benchmarks cost so much more (time, money, expertise)...
- timmaxw 11y ago
  > the author has corrected the discrepancies since then As of the time I posted this comment, the blog post still seems to be comparing indexed MongoDB operations against non-indexed RethinkDB operations. Under those conditions I'd expect RethinkDB to be at least 1000x slower than MongoDB. The fact that he's finding that RethinkDB is only 3x slower than MongoDB makes me think that there are still other major problems with this benchmark. > No, but at least he tried... It's true that the author tried; but that doesn't change the fact that people are going to read this blog post and assume that the numbers are at least approximately correct. As a RethinkDB employee, it really frustrates me to see RethinkDB being judged according to benchmarks that are conducted so carelessly that they are essentially random. I think this is the fourth time in the past year that I've seen a third party try to benchmark RethinkDB and get something wrong. Maybe we need to start a "best practice" of checking in with the maintainers of a project before publishing benchmark results about the project.
lukev 11y ago
This is why a lot of commercial databases have a "you may not publish benchmarks" clause in the license. It seems unfair and restrictive, but benchmarking is hard, and even where users get everything else right every data load is different. It's easy to see why companies don't want writeups like this one dominating searches for "<database> performance".
- userbinator 11y ago
  It seems even more unfair and restrictive to do that. If you prohibit benchmarks, it's essentially saying "we have something to hide" (like bad performance...), or "we don't want competition". I think it's far better to point out the flaws and specifics than to censor any attempt at comparison.
  - mejari 11y ago
    > it's essentially saying "we have something to hide" (like bad performance...), or "we don't want competition" This seems a totally imaginary dichotomy when literally the comment you're replying to presents an alternative option, namely; "We don't want you to publish things that are almost always going to be wrong and misleading"
mikecmpbll 11y ago
Any output like this, unless maliciously fallacious, is contributing in some way to the general understanding of the software concerned and benchmarking best-practices, even through its mistakes. It's the job of the reader to judge their sources wisely, and interrogate what they read, rather than the job of the author to conduct their explorations in private. Understandably, it can be frustrating for people involved in the projects but that's just the nature of the beast. They can do things to help their cause by championing good examples of benchmarking, even those which don't look upon them favourably.

165 points by neumino 11y ago

resonation 11y ago

...forgot to call run().

threeseed 11y ago
It was rerun with that call: https://www.amon.cx/blog/rethinkdb-reviewed-by-a-mongo-fan/ https://www.amon.cx/blog/rethinkdb-reviewed-by-a-mongo-fan/ Still we don't know the hardware, what versions were installed, configuration changes etc. The whole thing is pretty meaningless.

jakozaur 11y ago

Benchmarking is hard and a lot of reports are bogus. However they are still very useful for a lot of developers. Benchmarking programming languages got better. E.g.: http://benchmarksgame.alioth.debian.org/ http://benchmarksgame.alioth.debian.org/ gives roughly idea about performance of programing languages. I wish something similar existed for databases. I think exact figures would be hard to get, but I believe there are many 2x 10x differences that we should be aware of.

collyw 11y ago
In my experience, the database is the best place to perform optimizations, not in the code. I guess a lot depends on what you are doing.
amelius 11y ago
> Benchmarking programming languages got better. I wish we got this kind of benchmark for the altjs solutions.
boomlinde 11y ago
> Benchmarking is hard and a lot of reports are bogus. However they are still very useful for a lot of developers. In this case we were presented with a benchmark setup that failed to perform the task it supposedly benchmarked. That's not hard to avoid, and it makes the benchmark completely useless and misleading. I don't think that this problem can be generalized in the sense you do here, since the problem with benchmarks usually isn't a complete failure to perform the task to be benchmarked, but things like finding a set of tests that give a fair representation of what you'd typically use the subjects for, or performing the tasks in idiomatic and optimal ways.
amaranth 11y ago
Don't a lot of those benchmarks end up only measuring how fast your language can call out to GMP to do the real work? And regex-dna ends up measuring your regex implementation which for a lot of them is again just going to be measuring how fast they can all call out to PCRE. They're neat and all and it is called the benchmarks game but I wish they'd remove the ones that end up getting gamed like that.
- igouy 11y ago
  >> Don't a lot of those benchmarks… << No. >> …how fast they can all call out to PCRE. << No. Have you looked at the programs?

weavie 11y ago

> I expect to be able to click on a table and see the rows inside - like all the tools out there for Mongo, MySQL, PostgreSQL, etc. He does have a point there. It is slightly annoying having to type out a query when I just want to browse the data. (No real biggie though..)

CHY872 11y ago
It's not a good point, though. Writing a rudimentary tool of that type would take perhaps an hour.
- weavie 11y ago
  Indeed, and typing out the query really doesn't take that long. It does affect the initial experience though. The admin panel looks so slick I just assumed I would be able to click on the table and it would jump to the data. When it didn't I was surprised and initially blamed myself and fired up the console to see if there were any errors showing. It just affects the polish of the panel. As I said though it is a minor point for an excellent product.
neumino 11y ago
There are other tools for this, like Chateau: https://github.com/neumino/chateau https://github.com/neumino/chateau

tinussky 11y ago

benchmark = marketing (more times than not)

cpeterso 11y ago
Hence "benchmarketing".
threeseed 11y ago
Everyone likes to see how their tool of choice performs against other tools. And it's an important (albeit just one part) of deciding which product to use.
- tinussky 11y ago
  I always say do your own benchmarking for your own use case. The risk of a colored benchmark is quite high when benchmark is done by owner of product or by "fan" of product. With the exception of a well explained, clear benchmark that everyone can understand and reproduce easily.
  - davidw 11y ago
    At least with a public benchmark, people can point out flaws. With something rolled out internally, you're still likely to get flaws, but no one will point out that you misconfigured Postgres or set up Mongo the wrong way, or any of the other errors you are just as likely to make by doing it yourself. Perhaps, once you've winnowed the choices down to just a few, it might make more sense, but I think good public benchmarks can be a helpful thing for that selection process.
    - tinussky 11y ago
      I agree, public benchmarks do have their uses as i noted in my reply: "With the exception of a well explained, clear benchmark that everyone can understand and reproduce easily" An internal benchmark indeed requires knowledge on the subject, but most of the times people at the mailinglists are quite helpful when you explain what you are trying to do. Especially when you post a benchmark which is not in favorite of their product.

webtards 11y ago

Not sure what's worse here - people relying on third party benchmarks (hint: always do your own; see how a tool performs on your data, on your hardware, for your problem set), or the fanboy-ish panic when they are unsettled that a benchmark might make their chosen toy less shiny?

mikecmpbll 11y ago
The former :)

dantiberian 11y ago

I've been working with RethinkDB recently on some slightly unusual things and the Rethink team has been first class. They've got great support on IRC and GitHub and are open and friendly. I highly recommend them.

kalleboo 11y ago

From the linked benchmark: > In RethinDB, you have to create databases and tables manually and it will raise an exception if they already exist. Compared to MongoDB that could be an inconvenience for some(and me) - one of the things I find appealing in MongoDB is the fluid interaction with databases ... well at least now I don't feel so bad about having some old MySQL stuff still in production. MySQL already has too much "fluidity" in dealing with my data...

sklivvz1971 11y ago

This old comic piece seems to still apply: http://www.mongodb-is-web-scale.com/ http://www.mongodb-is-web-scale.com/

weddpros 11y ago
Then how on earth is it possible to build Parse.com with MongoDB? http://blog.parse.com/announcements/mongodb-rocksdb-parse/ http://blog.parse.com/announcements/mongodb-rocksdb-parse/
- sklivvz1971 11y ago
  Maybe you didn't get the joke. The joke is not about MongoDB, but about MongoDB fanbois that care only about some very narrow definition of "performance". The wider message is that DBs are way more complex beasts that is meaningful to test this way. Obviously there are cases in which MongoDB is a great choice, but equally obviously tests like this should not be a reason for the choice.
  - weddpros 11y ago
    I do understand that second degree, but I don't think the author is a fanboy... At work, we're also Mongodb users. If tomorrow we try to benchmark against Cassandra, performance will probably be the selling point. I don't think it's absurd to compare mongodb and rethinkdb, they're very similar dbs. As for the benchmark, I agree it doesn't explore every facet of both databases and focus on performance... That's what benchmarks do. The intent of the author may have been to challenge his existing choice (mongodb) which is a good thing. The (corrected) results may lead to: expect no performance gain if we migrate to RethinkDB... What's wrong with that?
    - sklivvz1971 11y ago
      Well, OK. I am not calling the author a fanboy, and i'll agree they probably aren't. There are at least 3 things here which invalidate this post because they make the numbers uncomparable: 1. Is the hardware adequate to run the tests, or does it favor one database? 2. Are both databases tuned to the use case? 3. Running only one type of action at a time is meaningless. Contention between reads and writes is what normally drives performance stats.

JDDunn9 11y ago

Would be nice to see some official benchmarks from RethinkDB. As this illustrates, there aren't any good ones out there.

insaneirish 11y ago
> Would be nice to see some official benchmarks from RethinkDB. Of some artificial scenario that people will claim does not reflect [their] reality? > As this illustrates, there aren't any good ones out there. The only good benchmarks are the ones that you define to be representative of something you'll actually see in reality.

weddpros 11y ago

Well, the author cares enough about RethinkDB to test it, even if he's a mongodb fan, even if his first benchmark was wrong, he was right to publish it: you all helped him when you pinpointed the problems in his tests... Thanks you for that. I don't see any marketing here, just the "do your own benchmark" best practice, and the "share with community" best practice... Does it make it a perfect benchmark? No, but at least he tried... and the author has corrected the discrepancies since then. Now imagine the benchmark was against [your favorite DB here] with even stronger results against RethinkDB. Notice how the most upvoted comment is joking about MongoDB. The second one is a pro-mysql comment. What's the point? Would it have been a better benchmark if it read "mysql is 10x faster than RethinkDB?" or "MongoDB is even slower than RethinkDB"?

sarciszewski 11y ago
I have to say, it's rare for me to agree with a HN comment as much as I do to this one. Well said!
sklivvz1971 11y ago
I suppose the author of the test forgot the "only test realistic scenarios" best practice.
- weddpros 11y ago
  We know nothing about the author's use case... Maybe he wanted to know where each DB shines compared to each other, to see if some workloads are better suited to one or the other. Of course, benchmarks "should" include concurrent reads/updates/writes/deletes because it can make a huge difference depending on the DB's implementation. Of course, the author "should" also have tested sharding / durability / resistance to partition / resource consumption in his tests... Maybe he didn't have the resources to test properly. I also do quick&dirty benchmarks like these, mostly because exhaustive benchmarks cost so much more (time, money, expertise)...
timmaxw 11y ago
> the author has corrected the discrepancies since then As of the time I posted this comment, the blog post still seems to be comparing indexed MongoDB operations against non-indexed RethinkDB operations. Under those conditions I'd expect RethinkDB to be at least 1000x slower than MongoDB. The fact that he's finding that RethinkDB is only 3x slower than MongoDB makes me think that there are still other major problems with this benchmark. > No, but at least he tried... It's true that the author tried; but that doesn't change the fact that people are going to read this blog post and assume that the numbers are at least approximately correct. As a RethinkDB employee, it really frustrates me to see RethinkDB being judged according to benchmarks that are conducted so carelessly that they are essentially random. I think this is the fourth time in the past year that I've seen a third party try to benchmark RethinkDB and get something wrong. Maybe we need to start a "best practice" of checking in with the maintainers of a project before publishing benchmark results about the project.

lukev 11y ago

This is why a lot of commercial databases have a "you may not publish benchmarks" clause in the license. It seems unfair and restrictive, but benchmarking is hard, and even where users get everything else right every data load is different. It's easy to see why companies don't want writeups like this one dominating searches for "<database> performance".

userbinator 11y ago
It seems even more unfair and restrictive to do that. If you prohibit benchmarks, it's essentially saying "we have something to hide" (like bad performance...), or "we don't want competition". I think it's far better to point out the flaws and specifics than to censor any attempt at comparison.
- mejari 11y ago
  > it's essentially saying "we have something to hide" (like bad performance...), or "we don't want competition" This seems a totally imaginary dichotomy when literally the comment you're replying to presents an alternative option, namely; "We don't want you to publish things that are almost always going to be wrong and misleading"

mikecmpbll 11y ago

Any output like this, unless maliciously fallacious, is contributing in some way to the general understanding of the software concerned and benchmarking best-practices, even through its mistakes. It's the job of the reader to judge their sources wisely, and interrogate what they read, rather than the job of the author to conduct their explorations in private. Understandably, it can be frustrating for people involved in the projects but that's just the nature of the beast. They can do things to help their cause by championing good examples of benchmarking, even those which don't look upon them favourably.