3 ms·

Well I have nothing constructive to add, but it certainly makes me rethink using MongoDB in any sort of serious production environment. I've been hesitant to us

Well I have nothing constructive to add, but it certainly makes me rethink using MongoDB in any sort of serious production environment. I've been hesitant to use MongoDB for this very reason...what happens when things go wrong? I'm certainly not an expert in how to handle those situations, and there aren't very many of those experts out there. Unfortunately for Foursquare, they got caught with their pants down, not having someone on staff who really understands all the ins & outs of a database technology their entire service depends upon.

dmytton 16y ago
You can get commercial support from 10gen (MongoDB developers) as we do for my company. It's just like MySQL providing support for their enterprise DB. Of course MySQL is much more widely used than MongoDB so there is more community help and knowledge.
- marclove 16y ago
  Right, I'm aware. Not everybody can afford a contract though, nor should they have to in order to avoid major outages like this. But I would like to know if Foursquare has a commercial support contract with 10Gen. If they didn't, why not? especially for a service that big? If they did, how was it that 10Gen took that long to fix the problem?
  - sachinag 16y ago
    They share USV as an investor. I'd be astonished if they didn't.
    - marclove 16y ago
      Heh...that's one hell of a commercial support contract then. Should be interesting to read 10gen's own detailed post-mortem.
  - steveklabnik 16y ago
    I'm sure that if you're the size of Foursquare, you can afford the contract. Then again, it's no excuse for bad software. I've only used Mongo on small sites so far, and have been loving it.
    - marclove 16y ago
      Its all fun and games until someone gets a shard in the eye.
  - fizx 16y ago
    I'm sure it was diagnosed quickly. Sometimes you have to copy data, rebuild indexes, etc.
    - fauigerzigerk 16y ago
      According to that blog post it hasn't actually been properly diagnosed yet. They see the symptoms but they don't know why they're seeing those symptoms.
  - cyu 16y ago
    If I was 10Gen, it would be in my best interest to offer support to 4sq, contract or not. People (like us) are watching.
rbranson 16y ago
Honestly though, would MySQL or PostgreSQL really helped out in this situation? Sharded or not sharded, there's really not much one can do once a server (or set of replicating servers comprising the shard) starts to become overloaded. Increasing the capacity of the shard by adding more hardware will induce a significant amount of load by itself. Of course, that's just one piece of the puzzle. We still don't know what actually brought the site down completely, hopefully they'll be able to trace it down and fill us in on that.
- Devilboy 16y ago
  The whole point of MongoDB is that you SHOULDN'T have to worry about a server becoming overloaded! You're giving up a lot for this privilege too, so if that doesn't even work properly then... back to PostgreSQL in my opinion.
  - rbranson 16y ago
    MongoDB is designed to help ease the pain of scaling, but what you're asking for is magic. If all the sudden a large amount of requests start coming through that overload a shard (think very popular users, like if several very popular users ended up colocated on a shard), how is MongoDB going to anticipate this? Any kind of shard scaling will only work well in scenarios where you have a reasonable increase in load, not a crippling surge. In addition, it's very, VERY difficult to scale writes against a single object, such as Justin Bieber's profile data, say, if you've got a view counter on it. You can either serialize writes on read like Cassandra does, which has it's own drawbacks (the more writers an object has, the more expensive reads become), or you can have single-master-for-an-object sharding like MongoDB employs and most other production sites (Facebook, Flickr, etc) use.
    - Devilboy 16y ago
      There was no sudden crippling surge in this case as far as I can tell. There was no mass-updating of a single object either. It failed even though it fits your ideal situation pretty much perfectly.
    - 16y ago

dmytton 16y ago

You can get commercial support from 10gen (MongoDB developers) as we do for my company. It's just like MySQL providing support for their enterprise DB. Of course MySQL is much more widely used than MongoDB so there is more community help and knowledge.

marclove 16y ago
Right, I'm aware. Not everybody can afford a contract though, nor should they have to in order to avoid major outages like this. But I would like to know if Foursquare has a commercial support contract with 10Gen. If they didn't, why not? especially for a service that big? If they did, how was it that 10Gen took that long to fix the problem?
- sachinag 16y ago
  They share USV as an investor. I'd be astonished if they didn't.
  - marclove 16y ago
    Heh...that's one hell of a commercial support contract then. Should be interesting to read 10gen's own detailed post-mortem.
- steveklabnik 16y ago
  I'm sure that if you're the size of Foursquare, you can afford the contract. Then again, it's no excuse for bad software. I've only used Mongo on small sites so far, and have been loving it.
  - marclove 16y ago
    Its all fun and games until someone gets a shard in the eye.
- fizx 16y ago
  I'm sure it was diagnosed quickly. Sometimes you have to copy data, rebuild indexes, etc.
  - fauigerzigerk 16y ago
    According to that blog post it hasn't actually been properly diagnosed yet. They see the symptoms but they don't know why they're seeing those symptoms.
- cyu 16y ago
  If I was 10Gen, it would be in my best interest to offer support to 4sq, contract or not. People (like us) are watching.

rbranson 16y ago

Honestly though, would MySQL or PostgreSQL really helped out in this situation? Sharded or not sharded, there's really not much one can do once a server (or set of replicating servers comprising the shard) starts to become overloaded. Increasing the capacity of the shard by adding more hardware will induce a significant amount of load by itself. Of course, that's just one piece of the puzzle. We still don't know what actually brought the site down completely, hopefully they'll be able to trace it down and fill us in on that.

Devilboy 16y ago
The whole point of MongoDB is that you SHOULDN'T have to worry about a server becoming overloaded! You're giving up a lot for this privilege too, so if that doesn't even work properly then... back to PostgreSQL in my opinion.
- rbranson 16y ago
  MongoDB is designed to help ease the pain of scaling, but what you're asking for is magic. If all the sudden a large amount of requests start coming through that overload a shard (think very popular users, like if several very popular users ended up colocated on a shard), how is MongoDB going to anticipate this? Any kind of shard scaling will only work well in scenarios where you have a reasonable increase in load, not a crippling surge. In addition, it's very, VERY difficult to scale writes against a single object, such as Justin Bieber's profile data, say, if you've got a view counter on it. You can either serialize writes on read like Cassandra does, which has it's own drawbacks (the more writers an object has, the more expensive reads become), or you can have single-master-for-an-object sharding like MongoDB employs and most other production sites (Facebook, Flickr, etc) use.
  - Devilboy 16y ago
    There was no sudden crippling surge in this case as far as I can tell. There was no mass-updating of a single object either. It failed even though it fits your ideal situation pretty much perfectly.
  - 16y ago