5 ms·

MongoDB 3.4 passes the rigorous and tough Jepsen test. Jepsen designs tests to make databases fail in terms of data consistency, correctness, and safety... Mong

MongoDB 3.4 passes the rigorous and tough Jepsen test. Jepsen designs tests to make databases fail in terms of data consistency, correctness, and safety... MongoDB 3.4 passed through their newest tests. I think that this really shows how mature of a Database MongoDB is.

siscia 10y ago
> In addition, we show that the new v1 replication protocol has multiple bugs, allowing data loss in all versions up to MongoDB 3.2.11 and 3.4.0-rc4. While the v0 protocol remains broken, patches for v1 are available in MongoDB 3.2.12 and 3.4.0, and now pass the expanded Jepsen test suite. Wait, it is me not understanding what the abstract is trying to say?
- wslh 10y ago
  I can only think that the article was a bit late and the new MongoDB version has the last word. Beyond this, the article is extremely valuable and it is mandatory to upgrade your MongoDB.
  - aphyr 10y ago
    This analysis was a collaboration with MongoDB over the past three months. MongoDB was able to fix these issues in 3.4.0 because I found them in rc3 and rc4.
    - wslh 10y ago
      Sorry, my fault.
- ealexhudson 10y ago
  It's saying it was fixed in the final 3.4.0 release (or, maybe, that there's at least a patch available) for the v1 protocol, but that the v0 protocol is unfixable - so, 3.4.0 + V1 protocol and it's solid. Giving aphyr some money to do this makes me think much more of Mongo now; I wouldn't previously have considered it for serious use but this is an excellent demonstration of intent.
untog 10y ago
> I think that this really shows how mature of a Database MongoDB is. Or that it took this long for them to pass basic proficiency tests. How do other database systems fare with these tests?
- jetpacktuxedo 10y ago
  You can check out the other databases he has tested here: http://jepsen.io/analyses http://jepsen.io/analyses That being said, none of the big players in sql are there, so you can't size it up against postgres or mysql.
  - mushi 10y ago
    isn't that because they are not distributed systems?
  - mat_keep 10y ago
    Galera was tested - enabling MySQL to behave like a distributed database
  - kedean 10y ago
    Postgres has been analyzed, I'm not sure why it's not on that list. https://aphyr.com/posts/282-call-me-maybe-postgres https://aphyr.com/posts/282-call-me-maybe-postgres
- Joeri 10y ago
  If you look at past Jepsen test mostly they fail, and sometimes they fail spectacularly. This is one of those things where if you don't have an automated test suite to continually verify correctness you will end up with bugs.
jhugg 10y ago
Note it’s not really “The Jepsen Test” as the tests are all a bit custom and different. Sometimes they’re looking to validate different claims. For example, Kyle integrated some serializability tests into the VoltDB Jepsen testing that wouldn’t apply to Mongo.
pvg 10y ago
It's great that they're serious and committing resources to passing these tests. They just did, though, after nearly a decade of development. Which is terrific but it doesn't really show the 'how mature of a database MongoDB is'.
aphyr 10y ago
While you are correct that 3.4 now passes, I feel like your interpretation here is a bit optimistic. It's not like I tested Mongo and it passed out of the box--it failed Jepsen tests, and not just with a read anomaly--it lost majority-acknowledged inserts. The v0 protocol still fails--it's fundamentally broken. The v1 bugs are fixed now, but that's a consequence of our collaboration. I'd say the marker of maturity here is that MongoDB has put significant time and effort into correctness: they take clock skew and network partitions as serious failure modes, they've redesigned their replication protocol, added options for stronger reads, and invested in their own correctness test suite, and Jepsen tests, as a part of their CI process.
- VeejayRampay 10y ago
  I'd like to use the occasion to thank you for the service you provide to the community. The tests themselves, the collaboration with vendors, the competition it fosters and the in-depth technical write-ups I think push the whole ecosystem forward, which everyone eventually benefits from. Much obliged.
  - aphyr 10y ago
    Thanks. :)
  - CJefferson 10y ago
    I agree -- I think the two most important people currently in software are aphyr for database testing, and lcamtuf for fuzzing (AFL). Both are pushing important fundamental software to pass the kind of tests it's embarrassing we haven't been doing for the past 20 years. I mean that in the most positive way, it's hard to get people to do things they've been neglecting for so long!
digitalzombie 10y ago
It's one test, paid for by MongoDB. It's good that they work with the dude. But there's only one test in town. I want to see more than just one test from more than one source and seeing their sketchy performance in the past only time will tell if this new version will be any better. > I think that this really shows how mature of a Database MongoDB is. Yeah no. It's a test. We have to wait and see in productions and see the data to prove that. It's also the first version to past this test. It means something but not a whole lot without real production data and other people attesting to it.

siscia 10y ago

> In addition, we show that the new v1 replication protocol has multiple bugs, allowing data loss in all versions up to MongoDB 3.2.11 and 3.4.0-rc4. While the v0 protocol remains broken, patches for v1 are available in MongoDB 3.2.12 and 3.4.0, and now pass the expanded Jepsen test suite. Wait, it is me not understanding what the abstract is trying to say?

wslh 10y ago
I can only think that the article was a bit late and the new MongoDB version has the last word. Beyond this, the article is extremely valuable and it is mandatory to upgrade your MongoDB.
- aphyr 10y ago
  This analysis was a collaboration with MongoDB over the past three months. MongoDB was able to fix these issues in 3.4.0 because I found them in rc3 and rc4.
  - wslh 10y ago
    Sorry, my fault.
ealexhudson 10y ago
It's saying it was fixed in the final 3.4.0 release (or, maybe, that there's at least a patch available) for the v1 protocol, but that the v0 protocol is unfixable - so, 3.4.0 + V1 protocol and it's solid. Giving aphyr some money to do this makes me think much more of Mongo now; I wouldn't previously have considered it for serious use but this is an excellent demonstration of intent.

untog 10y ago

> I think that this really shows how mature of a Database MongoDB is. Or that it took this long for them to pass basic proficiency tests. How do other database systems fare with these tests?

jetpacktuxedo 10y ago
You can check out the other databases he has tested here: http://jepsen.io/analyses http://jepsen.io/analyses That being said, none of the big players in sql are there, so you can't size it up against postgres or mysql.
- mushi 10y ago
  isn't that because they are not distributed systems?
- mat_keep 10y ago
  Galera was tested - enabling MySQL to behave like a distributed database
- kedean 10y ago
  Postgres has been analyzed, I'm not sure why it's not on that list. https://aphyr.com/posts/282-call-me-maybe-postgres https://aphyr.com/posts/282-call-me-maybe-postgres
Joeri 10y ago
If you look at past Jepsen test mostly they fail, and sometimes they fail spectacularly. This is one of those things where if you don't have an automated test suite to continually verify correctness you will end up with bugs.

jhugg 10y ago

Note it’s not really “The Jepsen Test” as the tests are all a bit custom and different. Sometimes they’re looking to validate different claims. For example, Kyle integrated some serializability tests into the VoltDB Jepsen testing that wouldn’t apply to Mongo.

pvg 10y ago

It's great that they're serious and committing resources to passing these tests. They just did, though, after nearly a decade of development. Which is terrific but it doesn't really show the 'how mature of a database MongoDB is'.

aphyr 10y ago

While you are correct that 3.4 now passes, I feel like your interpretation here is a bit optimistic. It's not like I tested Mongo and it passed out of the box--it failed Jepsen tests, and not just with a read anomaly--it lost majority-acknowledged inserts. The v0 protocol still fails--it's fundamentally broken. The v1 bugs are fixed now, but that's a consequence of our collaboration. I'd say the marker of maturity here is that MongoDB has put significant time and effort into correctness: they take clock skew and network partitions as serious failure modes, they've redesigned their replication protocol, added options for stronger reads, and invested in their own correctness test suite, and Jepsen tests, as a part of their CI process.

VeejayRampay 10y ago
I'd like to use the occasion to thank you for the service you provide to the community. The tests themselves, the collaboration with vendors, the competition it fosters and the in-depth technical write-ups I think push the whole ecosystem forward, which everyone eventually benefits from. Much obliged.
- aphyr 10y ago
  Thanks. :)
- CJefferson 10y ago
  I agree -- I think the two most important people currently in software are aphyr for database testing, and lcamtuf for fuzzing (AFL). Both are pushing important fundamental software to pass the kind of tests it's embarrassing we haven't been doing for the past 20 years. I mean that in the most positive way, it's hard to get people to do things they've been neglecting for so long!

digitalzombie 10y ago

It's one test, paid for by MongoDB. It's good that they work with the dude. But there's only one test in town. I want to see more than just one test from more than one source and seeing their sketchy performance in the past only time will tell if this new version will be any better. > I think that this really shows how mature of a Database MongoDB is. Yeah no. It's a test. We have to wait and see in productions and see the data to prove that. It's also the first version to past this test. It means something but not a whole lot without real production data and other people attesting to it.