3 ms·

Single DB corruption and disk failures are more common than you think. "Single master, replicated to another region." Why do you recommend against using a mod

by _vvhw 5y ago

Single DB corruption and disk failures are more common than you think. "Single master, replicated to another region." Why do you recommend against using a modern distributed database? Is it that hard to spin up something like CockroachDB? In fact, it wouldn't surprise me that Wave are running CockroachDB as their Postgres foundation under the hood.

lkrubner 5y ago
> Why do you recommend against using a modern distributed database? Call me maybe. This is a newer technology and the growing pains have been dramatic. I assume you've followed Jespen? https://aphyr.com/tags/jepsen https://aphyr.com/tags/jepsen I use MongoDB all the time, for a lot of projects. But I'm also keenly aware of how many years they have struggled to pass the Jespen tests. I can also recall several times engineers from MongoDB have said "We have finally resolved all of the issues raised by Jespen" and then another bug was discovered. Again, I use MongoDB all the time. But I'm aware it is simpler and safer to use Postgres, for many, many use cases.
- _vvhw 5y ago
  "I assume you've followed Jespen?" Yes! You might also want to check out autonomous deterministic testing, which is like Jepsen, but where you can explore many more state spaces by speeding up time, and where you can replay any distributed bugs instantly, just by replaying a seed. This is a brilliant talk on these techniques: FoundationDB — How I Learned to Stop Worrying and Trust the Database — https://www.youtube.com/watch?v=OJb8A6h9jQQ https://www.youtube.com/watch?v=OJb8A6h9jQQ I also work on a distributed database called TigerBeetle [1] where we use these new techniques. You can run our simulation tests yourself even. It's as simple as cloning the repo and running "scripts/vopr.sh", and let me know if you want a tour! These testing techniques work so well that we've been running a bug bounty challenge with awards up to $8192 for anyone who can find a way to break it or crash it. It comes with the simulation testing all included so it's pretty easy to get started. [2] [1] https://github.com/coilhq/tigerbeetle https://github.com/coilhq/tigerbeetle [2] https://github.com/coilhq/viewstamped-replication-made-famous https://github.com/coilhq/viewstamped-replication-made-famou...
- jd_mongodb 5y ago
  I'm not sure "struggled to pass" is the right characterisation. The Jepsen tests are part of the MongoDB test suite. They have found some bugs in the past and we have fixed them and incorporated into our test suite. Am I about to say Jepsen will never find another bug in MongoDB? No. But if they do we will fix those as well.
  - _vvhw 5y ago
    And I understand that MongoDB are also testing with Antithesis [1] and getting lots of assurance from that, right? That gives me confidence. [1] https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-A...
  - lkrubner 5y ago
    I followed MongoDB and Jespen in detail for years and I think "struggle to pass" is a correct characterization, which I believe even Kyle Kingsbury used it at one point. There was a long stretch where MongoDB treated Kingsbury as an enemy, and simply ignored him, or said he was wrong, then there were some years where they were saying "we dealt with all of the issues that Kingsbury raised" and then Kingsbury would write another blog post saying "Well, I still find these issues." Finally, MongoDB changed gears and decided to work with Kingsbury in a productive way, and then Jespen began to become part of the MongoDB test suite.

by _vvhw 5y ago

lkrubner 5y ago

> Why do you recommend against using a modern distributed database? Call me maybe. This is a newer technology and the growing pains have been dramatic. I assume you've followed Jespen? https://aphyr.com/tags/jepsen https://aphyr.com/tags/jepsen I use MongoDB all the time, for a lot of projects. But I'm also keenly aware of how many years they have struggled to pass the Jespen tests. I can also recall several times engineers from MongoDB have said "We have finally resolved all of the issues raised by Jespen" and then another bug was discovered. Again, I use MongoDB all the time. But I'm aware it is simpler and safer to use Postgres, for many, many use cases.

_vvhw 5y ago
"I assume you've followed Jespen?" Yes! You might also want to check out autonomous deterministic testing, which is like Jepsen, but where you can explore many more state spaces by speeding up time, and where you can replay any distributed bugs instantly, just by replaying a seed. This is a brilliant talk on these techniques: FoundationDB — How I Learned to Stop Worrying and Trust the Database — https://www.youtube.com/watch?v=OJb8A6h9jQQ https://www.youtube.com/watch?v=OJb8A6h9jQQ I also work on a distributed database called TigerBeetle [1] where we use these new techniques. You can run our simulation tests yourself even. It's as simple as cloning the repo and running "scripts/vopr.sh", and let me know if you want a tour! These testing techniques work so well that we've been running a bug bounty challenge with awards up to $8192 for anyone who can find a way to break it or crash it. It comes with the simulation testing all included so it's pretty easy to get started. [2] [1] https://github.com/coilhq/tigerbeetle https://github.com/coilhq/tigerbeetle [2] https://github.com/coilhq/viewstamped-replication-made-famous https://github.com/coilhq/viewstamped-replication-made-famou...
jd_mongodb 5y ago
I'm not sure "struggled to pass" is the right characterisation. The Jepsen tests are part of the MongoDB test suite. They have found some bugs in the past and we have fixed them and incorporated into our test suite. Am I about to say Jepsen will never find another bug in MongoDB? No. But if they do we will fix those as well.
- _vvhw 5y ago
  And I understand that MongoDB are also testing with Antithesis [1] and getting lots of assurance from that, right? That gives me confidence. [1] https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-Antithesis https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-A...
- lkrubner 5y ago
  I followed MongoDB and Jespen in detail for years and I think "struggle to pass" is a correct characterization, which I believe even Kyle Kingsbury used it at one point. There was a long stretch where MongoDB treated Kingsbury as an enemy, and simply ignored him, or said he was wrong, then there were some years where they were saying "we dealt with all of the issues that Kingsbury raised" and then Kingsbury would write another blog post saying "Well, I still find these issues." Finally, MongoDB changed gears and decided to work with Kingsbury in a productive way, and then Jespen began to become part of the MongoDB test suite.