6 ms·
Interestingly the cause of the most severe part of their downtime seems to still be unknown to them: As a next step, we introduced a new shard, intending to mo
by brown9-2 16y ago
Interestingly the cause of the most severe part of their downtime seems to still be unknown to them:
As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one.
We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go down.
To those who use MongoDB - does this sound like something that might have been caused by MongoDB itself, or Foursquare's use of it?
- houseabsolute 16y agoI'm also wondering why this kind of intervention is necessary at all. The NoSQL solution we use at work has load based automatic splitting, and I'd have thought (though I haven't confirmed) that this would be an obvious feature to include.
- icey 16y agoWhich NoSQL store do you use?
- houseabsolute 16y agohttp://labs.google.com/papers/bigtable.html http://labs.google.com/papers/bigtable.html
- staunch 16y agoThat looks potentially useful in some cases. Can you send me the code/docs for that?
- ddlatham 16y agohttp://hbase.apache.org/ http://hbase.apache.org/
- ifesdjeen 16y agoDoes it work that well for production for you? Really, that's extremely interesting.
- rb2k_ 16y agoAlso look at dynamo based systems like cassandra and riak (riak seems to have a better load balancing at the moment, cassandra is a bit more "bumpy")
- dmytton 16y agoLoad based splitting is on the MongoDB roadmap, but doesn't exist yet.
- fizx 16y agoJust because the feature exists, doesn't mean it works.
- mcfunley 16y agoI would speculate that it's a poorly-chosen shard key. MongoDB's built-in sharding uses range-based indexing. If you choose user_id as your shard key, and those are autoincrementing integers, then you're screwed if newer users tend to be more active on average than older ones.
- jrockway 16y agoWait, people shard on a key other than something approximately random, like an sha1 hash!?
- mcfunley 16y agoWhere I work (Etsy) we keep an index server that maps each user to a shard on an individual basis. There are a number of advantages to it. For example, if one user generated a ton of activity they could in theory be moved to their own server. Approximately random works for the initial assignment. Flickr works the same way (not by coincidence, since we have several former Flickr engineers on staff).
- jrockway 16y agoSounds like a good system. I've noticed that people tend to do things like shard based on even/odd, and then they realize that they need three databases. I've never had either problem though... but if I ever need to shard I plan on doing it based on object ID. Then one request can be handled by multiple databases, "for free", increasing both throughput and response time.
- fizx 16y agoEven/odd isn't the end of the world, but you would then be best jumping to mod 4.
- moe 16y agoActually for anonymous sharding (without a central index) a consistent hash is about the closest you can get to ideal distribution and flexibility. I haven't looked but I presume that's what mongo uses under the hood for their auto-sharding, too.
- dmytton 16y agoWithout knowing more about their schema, it's hard to speculate. However, let's assume they had a single, large "checkins" collection, sharded by user ID as they said. That would be distributed across the shards. If the mongod process is down on one shard and therefore the shard isn't responding, the data just wouldn't be available. But it sounds like the shard was still up. As per the sharding FAQ[1]: "What if a shard is down or slow and I do a query? If a shard is down, the query will return an error. If a shard is responding slowly, mongos will wait for it. You won't get partial results." They said they had a performance issue on the overloaded shard so perhaps it's possible that mongo believed the shard to be up, when it was instead overloaded. This meant any queries just waited, taking down the whole site. As I said, this is speculation and as a massive production user of MongoDB ourselves[2], I'm interested to know more. [1] http://www.mongodb.org/display/DOCS/Sharding+FAQ http://www.mongodb.org/display/DOCS/Sharding+FAQ [2] http://blog.boxedice.com/2010/02/28/notes-from-a-production-mongodb-deployment/ http://blog.boxedice.com/2010/02/28/notes-from-a-production-...
- marclove 16y agoBut it wasn't the overloaded shard that brought the site down (it was only slowing things down), it was the addition of the new shard: "For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go down."
- dmytton 16y agoIf the data was being moved it could cause even more load on an already overloaded shard.
- VladRussian 16y ago"What if a shard is down or slow and I do a query? If a shard is down, the query will return an error. If a shard is responding slowly, mongos will wait for it. You won't get partial results." an approach to CAP that chooses consistency over availability. I wonder why as it isn't a bank and they could have chosen the availability instead.
- scg 16y agoAt 6:30pm EST, we determined the most effective course of action was to re-index the shard, which would address the memory fragmentation and usage issues. Sounds similar to some of the problems we're facing with MongoDB too. Indexes fragment a lot, especially if you delete documents frequently. You have to reindex everything eventually. MongoDB offers background indexing (except for _id fields!) but it's not really an option if you need the indexes at all times to keep executing queries. Also, you'll need to reindex _id eventually and there's currently no way to do that without blocking the server.
- rb2k_ 16y ago> To those who use MongoDB - does this sound like something that might have been caused by MongoDB itself, or Foursquare's use of it? MongoDB still has problems with locking too much for certain operations. The 10gen guys keep on introducing "yields" to those operations, but the simple (and fast) database/process-level locking could very well cause this... (not a 100% 'yes', but it seems possible)