6 ms·

Retrospective from Postmark on outages (MongoDB)

46 points by cemerick 15y ago

viraptor 15y ago
I get the impression that one of the biggest issues was missed. They did not test the standard load against the secondary server, they assigned a machine of lower specs to the task and there's nothing in the future actions that indicates they'll change it... Even if they go for the new and shiny, they can end up in the same situation when their master fails. I hope they just overlooked that in the blog post, rather than actually not correcting this first.
- mtkd 15y ago
  It's also advisable to try and keep the core dataset (which you absolutely depend on) as light as possible. Split out the heavy stuff on to other servers. Then have emergency flags in the webapps so you can run them in a low feature mode. If you bake this concept in when you're building the webapps dealing with drama is much less stressful.
- calpaterson 15y ago
  This seems like such an obvious issue that I can't understand how they overlooked it. If you are using failover as a strategy, your failover machine has to be up to the task.
- steve8918 15y ago
  That's my takeaway as well. I have no opinions on MongoDB, but it really seems like this particular problem was because they skimped on disaster recovery, ie. their failover hardware was less powerful than their production hardware. That was the root cause of their downtime, which is inadequate planning. That's spending money on car insurance, but realizing only after you get into an accident that the car insurance covers almost nothing. It means you've wasted your money paying for the insurance. They paid for the secondary failover hardware, but it was effectively useless since they were down for 2 days. The only thing it mitigated, possibly, was how long they were down for, but the primary objective of the hardware, ie. keep them up in case of a disaster, was a complete failure. I've worked at a company that was completely down for a day worldwide due to a "disaster", even though we had spent millions on diesel fuel generators, etc. I blame the "checkbox" mentality where people only look to satisfy requirements, but no one actually has ownership over the process and the details. Unfortunately, in my case, no one got fired over this complete misstep, which is another problem... zero accountability.
  - viraptor 15y ago
    Seems like Netflix's chaos monkey is not a bad idea actually. I don't mean you have to kill your services randomly while there are users on them... but switching from your master to secondary (why are you even making a distinction anyway?) should be a pretty standard operation. Even normal upgrades (hardware fails - it's a question of when, not if) could be handled transparently just by making the "secondary" server a first-class citizen.
megaman821 15y ago
I know I shouldn't care but I am always weary of doing business with any company using MongoDB. 10gen put out claiming MongoDB was so much faster than SQL solutions but it seemed obvious to me than turning off fsync would make the SQL solutions run at about the same speed. Plus, why would I want to run my database in mode where it is easy to lose data? MongoDB may be a useful product but their marketing is deceptive, which will lead to companies using it in inappropriate situations.
- rogerbinns 15y ago
  Have you noticed that the problem isn't MongoDB? Every case I've seen has been the data traffic exceeding the capacity of the system. There is no database that works well under those circumstances. Another way of putting it is that these businesses have been sufficiently successful that they have outgrown their own planning and deployment. As for your second paragraph, MongoDB has had journalling for quite a while, so you can make you writes durable and limited to the speed of your storage.
  - thibaut_barrere 15y ago
    > Every case I've seen has been the data traffic exceeding the capacity of the system. There is no database that works well under those circumstances. Is there a benchmark somewhere comparing the memory/disk consumption of MongoDB vs. other datastores? If there's a significant overhead (and my early tests tend to show that there was - but I didn't make a strict benchmark though), then it would become very related to MongoDB then. (honest and real question, I'm a MongoDB user btw, as well as Redis, MySQL, Postgresql etc).
    - rogerbinns 15y ago
      The main overhead for MongoDB's storage is that the "column name" (keys) is stored in every record rather than just once as with traditional SQL database and some of the other NoSQL solutions. That is why you'll often see developers using very short key names, and one use for an "ORM" to translate between developer friendly names and short stored names. Of course this can be solved fairly easily by the MongoDB developers by having a table mapping between short tokens/numbers and the long names. This is the ticket: https://jira.mongodb.org/browse/SERVER-863 https://jira.mongodb.org/browse/SERVER-863 This is someone's measurements with different key names: http://christophermaier.name/blog/2011/05/22/MongoDB-key-names http://christophermaier.name/blog/2011/05/22/MongoDB-key-nam...
brainless 15y ago
Are they using MongoDB for the wrong purposes? The task they have seems similar to logging and if so aren't there much better software to do that? A logging server perhaps?
- rogerbinns 15y ago
  MongoDB does have what it calls capped collections - the semantics are the same as a circular log. No idea why they don't use that storage format. In any event this problem is one of success. That is the kind of problem I prefer having.
reustle 15y ago
So the primary was much beefier than the secondary? If so, why let it fail over? I figured you should always have the same resources for a mongo primary and secondary.
ninjastar99 15y ago
Really tried to love Postmark for about 3 months now. Constant, almost daily issues forced us to unfollow them on Twitter (it literally became an annoyance seeing issues daily). Then we switched to Mailgun last week, and we are very happy. +1 to Mailgun
bsg75 15y ago
Two is one. One is none. This includes equality in failover systems. The common issue in all of these cases is not the DB engine, OS, stack, whatever, but the infrastructure.

46 points by cemerick 15y ago

I get the impression that one of the biggest issues was missed. They did not test the standard load against the secondary server, they assigned a machine of lower specs to the task and there's nothing in the future actions that indicates they'll change it... Even if they go for the new and shiny, they can end up in the same situation when their master fails. I hope they just overlooked that in the blog post, rather than actually not correcting this first.

mtkd 15y ago
It's also advisable to try and keep the core dataset (which you absolutely depend on) as light as possible. Split out the heavy stuff on to other servers. Then have emergency flags in the webapps so you can run them in a low feature mode. If you bake this concept in when you're building the webapps dealing with drama is much less stressful.
calpaterson 15y ago
This seems like such an obvious issue that I can't understand how they overlooked it. If you are using failover as a strategy, your failover machine has to be up to the task.
steve8918 15y ago
That's my takeaway as well. I have no opinions on MongoDB, but it really seems like this particular problem was because they skimped on disaster recovery, ie. their failover hardware was less powerful than their production hardware. That was the root cause of their downtime, which is inadequate planning. That's spending money on car insurance, but realizing only after you get into an accident that the car insurance covers almost nothing. It means you've wasted your money paying for the insurance. They paid for the secondary failover hardware, but it was effectively useless since they were down for 2 days. The only thing it mitigated, possibly, was how long they were down for, but the primary objective of the hardware, ie. keep them up in case of a disaster, was a complete failure. I've worked at a company that was completely down for a day worldwide due to a "disaster", even though we had spent millions on diesel fuel generators, etc. I blame the "checkbox" mentality where people only look to satisfy requirements, but no one actually has ownership over the process and the details. Unfortunately, in my case, no one got fired over this complete misstep, which is another problem... zero accountability.
- viraptor 15y ago
  Seems like Netflix's chaos monkey is not a bad idea actually. I don't mean you have to kill your services randomly while there are users on them... but switching from your master to secondary (why are you even making a distinction anyway?) should be a pretty standard operation. Even normal upgrades (hardware fails - it's a question of when, not if) could be handled transparently just by making the "secondary" server a first-class citizen.

megaman821 15y ago

I know I shouldn't care but I am always weary of doing business with any company using MongoDB. 10gen put out claiming MongoDB was so much faster than SQL solutions but it seemed obvious to me than turning off fsync would make the SQL solutions run at about the same speed. Plus, why would I want to run my database in mode where it is easy to lose data? MongoDB may be a useful product but their marketing is deceptive, which will lead to companies using it in inappropriate situations.

rogerbinns 15y ago
Have you noticed that the problem isn't MongoDB? Every case I've seen has been the data traffic exceeding the capacity of the system. There is no database that works well under those circumstances. Another way of putting it is that these businesses have been sufficiently successful that they have outgrown their own planning and deployment. As for your second paragraph, MongoDB has had journalling for quite a while, so you can make you writes durable and limited to the speed of your storage.
- thibaut_barrere 15y ago
  > Every case I've seen has been the data traffic exceeding the capacity of the system. There is no database that works well under those circumstances. Is there a benchmark somewhere comparing the memory/disk consumption of MongoDB vs. other datastores? If there's a significant overhead (and my early tests tend to show that there was - but I didn't make a strict benchmark though), then it would become very related to MongoDB then. (honest and real question, I'm a MongoDB user btw, as well as Redis, MySQL, Postgresql etc).
  - rogerbinns 15y ago
    The main overhead for MongoDB's storage is that the "column name" (keys) is stored in every record rather than just once as with traditional SQL database and some of the other NoSQL solutions. That is why you'll often see developers using very short key names, and one use for an "ORM" to translate between developer friendly names and short stored names. Of course this can be solved fairly easily by the MongoDB developers by having a table mapping between short tokens/numbers and the long names. This is the ticket: https://jira.mongodb.org/browse/SERVER-863 https://jira.mongodb.org/browse/SERVER-863 This is someone's measurements with different key names: http://christophermaier.name/blog/2011/05/22/MongoDB-key-names http://christophermaier.name/blog/2011/05/22/MongoDB-key-nam...

brainless 15y ago

Are they using MongoDB for the wrong purposes? The task they have seems similar to logging and if so aren't there much better software to do that? A logging server perhaps?

rogerbinns 15y ago
MongoDB does have what it calls capped collections - the semantics are the same as a circular log. No idea why they don't use that storage format. In any event this problem is one of success. That is the kind of problem I prefer having.

reustle 15y ago

So the primary was much beefier than the secondary? If so, why let it fail over? I figured you should always have the same resources for a mongo primary and secondary.

ninjastar99 15y ago

Really tried to love Postmark for about 3 months now. Constant, almost daily issues forced us to unfollow them on Twitter (it literally became an annoyance seeing issues daily). Then we switched to Mailgun last week, and we are very happy. +1 to Mailgun

bsg75 15y ago

Two is one. One is none. This includes equality in failover systems. The common issue in all of these cases is not the DB engine, OS, stack, whatever, but the infrastructure.