46 ms·

What do you mean by "troubled," and what docs did you follow to set up and run the cluster? Caring and feeding for a database (of any type) with any type of HA

What do you mean by "troubled," and what docs did you follow to set up and run the cluster? Caring and feeding for a database (of any type) with any type of HA has a learning curve. Hence the growing number of PaaS services that handle the setup and maintenance for you (AWS's RDS, Mongo's Atlas, etc)

cj 6y ago
MongoDB's own management software (Cloud Manager, Ops Manager) is littered with bugs to the point where it's nearly unusable for certain actions. (One of which being restoring backups) It was really, really bad 3-4 years ago. Regularly entering irrecoverable error states while performing basic management operations via MongoDB's management GUI. I've noticed a significant improvement in the past ~1 year.
ashtonkem 6y ago
By troubled I mean it would regularly go into what I called "three stooges" mode where each node swore that another was the master. Of course this would mean that writes would stop dead in their tracks. Given that this happened to me across companies, teams, and a half decade of time, I've decided that this is a case where the problem is Mongo and not me.
- ryanianian 6y ago
  The only constant there is you which means it is perhaps you. Setting up and caring for a DB cluster is a complicated thing, to the point of there being non-BS certification courses for nearly every major HA database including Mongo. It very well could be that there was a well-documented flag that you never learned. This doesn't mean you're being unreasonable. I'd be cranky with any DB getting itself in a split-brain scenario, but my conclusion wouldn't be a bug in the software but rather it's a bug in my understanding. It's worth noting that getting in this state should either be impossible, or it should be obvious about how it arrived at such a state with links to relevant documentation. (There's also little incentive to make it super easy to run in production on your own. They sell that as a service after all. It wouldn't surprise me if the product had invariants that assumed production-level configurations and that nobody's tested with whatever configs ended up making it go nuts.)
  - jokethrowaway 6y ago
    I share the experience of parent. My startup ages ago wasn't big enough to have perf issues with mongo, but I certainly experienced data loss (ore wiredtiger). A company I worked for (with the most capable team of DevOps I've seen in my career, incidentally) eventually moved from self hosting mongo to atlas, hoping to ease the pain. This was already a few years after wiredtiger came out and fix a lot of issues. Performance problems kept happening and we just started switching to postgres. At my new job, DevOps people keep complaining about our mongo clusters on atlas. It's again, terrible perf and occasional weird issues. We did a disk upgrade and the cluster took 3 days to move all the replica from starting up (and replicating data) to running. During those three days we were running with one less replica and everything was to slow. A complete nightmare. Adding another replica wouldn't have solved anything because it would have taken a day to finish booting. They've been trying to migrate off mongo for years but there is never time, as usual.

cj 6y ago

MongoDB's own management software (Cloud Manager, Ops Manager) is littered with bugs to the point where it's nearly unusable for certain actions. (One of which being restoring backups) It was really, really bad 3-4 years ago. Regularly entering irrecoverable error states while performing basic management operations via MongoDB's management GUI. I've noticed a significant improvement in the past ~1 year.

ashtonkem 6y ago

By troubled I mean it would regularly go into what I called "three stooges" mode where each node swore that another was the master. Of course this would mean that writes would stop dead in their tracks. Given that this happened to me across companies, teams, and a half decade of time, I've decided that this is a case where the problem is Mongo and not me.

ryanianian 6y ago
The only constant there is you which means it is perhaps you. Setting up and caring for a DB cluster is a complicated thing, to the point of there being non-BS certification courses for nearly every major HA database including Mongo. It very well could be that there was a well-documented flag that you never learned. This doesn't mean you're being unreasonable. I'd be cranky with any DB getting itself in a split-brain scenario, but my conclusion wouldn't be a bug in the software but rather it's a bug in my understanding. It's worth noting that getting in this state should either be impossible, or it should be obvious about how it arrived at such a state with links to relevant documentation. (There's also little incentive to make it super easy to run in production on your own. They sell that as a service after all. It wouldn't surprise me if the product had invariants that assumed production-level configurations and that nobody's tested with whatever configs ended up making it go nuts.)
- jokethrowaway 6y ago
  I share the experience of parent. My startup ages ago wasn't big enough to have perf issues with mongo, but I certainly experienced data loss (ore wiredtiger). A company I worked for (with the most capable team of DevOps I've seen in my career, incidentally) eventually moved from self hosting mongo to atlas, hoping to ease the pain. This was already a few years after wiredtiger came out and fix a lot of issues. Performance problems kept happening and we just started switching to postgres. At my new job, DevOps people keep complaining about our mongo clusters on atlas. It's again, terrible perf and occasional weird issues. We did a disk upgrade and the cluster took 3 days to move all the replica from starting up (and replicating data) to running. During those three days we were running with one less replica and everything was to slow. A complete nightmare. Adding another replica wouldn't have solved anything because it would have taken a day to finish booting. They've been trying to migrate off mongo for years but there is never time, as usual.