8 ms·

So, that was a bummer

189 points by abraham 16y ago

blantonl 16y ago
we noticed that one of these shards was performing poorly because a disproportionate share of check-ins were being written to it. I'd love to know the root cause behind this specific issue. Was this a behavioral issue within the user base, or a technical problem that routed check-ins to this specific shard more than others? Since they mentioned they partition their shards by userId that would probably rule out their routing process. I wonder if there was some event that caused a certain sharded subsection of users to start sending so many checkins? And since this was a subset of userIds on the same shard - could this have been a targeted DOS or SPAM event? I'm making a very conscious migration to MongoDB so I'm very interested to hear what the root cause of this was.
brown9-2 16y ago
Interestingly the cause of the most severe part of their downtime seems to still be unknown to them: As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one. We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go down. To those who use MongoDB - does this sound like something that might have been caused by MongoDB itself, or Foursquare's use of it?
- houseabsolute 16y ago
  I'm also wondering why this kind of intervention is necessary at all. The NoSQL solution we use at work has load based automatic splitting, and I'd have thought (though I haven't confirmed) that this would be an obvious feature to include.
  - icey 16y ago
    Which NoSQL store do you use?
    - houseabsolute 16y ago
      http://labs.google.com/papers/bigtable.html http://labs.google.com/papers/bigtable.html
      - staunch 16y ago
        That looks potentially useful in some cases. Can you send me the code/docs for that?
        ddlatham 16y ago
        http://hbase.apache.org/ http://hbase.apache.org/
        ifesdjeen 16y ago
        Does it work that well for production for you? Really, that's extremely interesting.
        rb2k_ 16y ago
        Also look at dynamo based systems like cassandra and riak (riak seems to have a better load balancing at the moment, cassandra is a bit more "bumpy")
  - dmytton 16y ago
    Load based splitting is on the MongoDB roadmap, but doesn't exist yet.
  - fizx 16y ago
    Just because the feature exists, doesn't mean it works.
  - mcfunley 16y ago
    I would speculate that it's a poorly-chosen shard key. MongoDB's built-in sharding uses range-based indexing. If you choose user_id as your shard key, and those are autoincrementing integers, then you're screwed if newer users tend to be more active on average than older ones.
    - jrockway 16y ago
      Wait, people shard on a key other than something approximately random, like an sha1 hash!?
      - mcfunley 16y ago
        Where I work (Etsy) we keep an index server that maps each user to a shard on an individual basis. There are a number of advantages to it. For example, if one user generated a ton of activity they could in theory be moved to their own server. Approximately random works for the initial assignment. Flickr works the same way (not by coincidence, since we have several former Flickr engineers on staff).
        jrockway 16y ago
        Sounds like a good system. I've noticed that people tend to do things like shard based on even/odd, and then they realize that they need three databases. I've never had either problem though... but if I ever need to shard I plan on doing it based on object ID. Then one request can be handled by multiple databases, "for free", increasing both throughput and response time.
        fizx 16y ago
        Even/odd isn't the end of the world, but you would then be best jumping to mod 4.
        moe 16y ago
        Actually for anonymous sharding (without a central index) a consistent hash is about the closest you can get to ideal distribution and flexibility. I haven't looked but I presume that's what mongo uses under the hood for their auto-sharding, too.
- dmytton 16y ago
  Without knowing more about their schema, it's hard to speculate. However, let's assume they had a single, large "checkins" collection, sharded by user ID as they said. That would be distributed across the shards. If the mongod process is down on one shard and therefore the shard isn't responding, the data just wouldn't be available. But it sounds like the shard was still up. As per the sharding FAQ[1]: "What if a shard is down or slow and I do a query? If a shard is down, the query will return an error. If a shard is responding slowly, mongos will wait for it. You won't get partial results." They said they had a performance issue on the overloaded shard so perhaps it's possible that mongo believed the shard to be up, when it was instead overloaded. This meant any queries just waited, taking down the whole site. As I said, this is speculation and as a massive production user of MongoDB ourselves[2], I'm interested to know more. [1] http://www.mongodb.org/display/DOCS/Sharding+FAQ http://www.mongodb.org/display/DOCS/Sharding+FAQ [2] http://blog.boxedice.com/2010/02/28/notes-from-a-production-mongodb-deployment/ http://blog.boxedice.com/2010/02/28/notes-from-a-production-...
  - marclove 16y ago
    But it wasn't the overloaded shard that brought the site down (it was only slowing things down), it was the addition of the new shard: "For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go down."
    - dmytton 16y ago
      If the data was being moved it could cause even more load on an already overloaded shard.
  - VladRussian 16y ago
    "What if a shard is down or slow and I do a query? If a shard is down, the query will return an error. If a shard is responding slowly, mongos will wait for it. You won't get partial results." an approach to CAP that chooses consistency over availability. I wonder why as it isn't a bank and they could have chosen the availability instead.
- scg 16y ago
  At 6:30pm EST, we determined the most effective course of action was to re-index the shard, which would address the memory fragmentation and usage issues. Sounds similar to some of the problems we're facing with MongoDB too. Indexes fragment a lot, especially if you delete documents frequently. You have to reindex everything eventually. MongoDB offers background indexing (except for _id fields!) but it's not really an option if you need the indexes at all times to keep executing queries. Also, you'll need to reindex _id eventually and there's currently no way to do that without blocking the server.
- rb2k_ 16y ago
  > To those who use MongoDB - does this sound like something that might have been caused by MongoDB itself, or Foursquare's use of it? MongoDB still has problems with locking too much for certain operations. The 10gen guys keep on introducing "yields" to those operations, but the simple (and fast) database/process-level locking could very well cause this... (not a 100% 'yes', but it seems possible)
marclove 16y ago
Well I have nothing constructive to add, but it certainly makes me rethink using MongoDB in any sort of serious production environment. I've been hesitant to use MongoDB for this very reason...what happens when things go wrong? I'm certainly not an expert in how to handle those situations, and there aren't very many of those experts out there. Unfortunately for Foursquare, they got caught with their pants down, not having someone on staff who really understands all the ins & outs of a database technology their entire service depends upon.
- dmytton 16y ago
  You can get commercial support from 10gen (MongoDB developers) as we do for my company. It's just like MySQL providing support for their enterprise DB. Of course MySQL is much more widely used than MongoDB so there is more community help and knowledge.
  - marclove 16y ago
    Right, I'm aware. Not everybody can afford a contract though, nor should they have to in order to avoid major outages like this. But I would like to know if Foursquare has a commercial support contract with 10Gen. If they didn't, why not? especially for a service that big? If they did, how was it that 10Gen took that long to fix the problem?
    - sachinag 16y ago
      They share USV as an investor. I'd be astonished if they didn't.
      - marclove 16y ago
        Heh...that's one hell of a commercial support contract then. Should be interesting to read 10gen's own detailed post-mortem.
    - steveklabnik 16y ago
      I'm sure that if you're the size of Foursquare, you can afford the contract. Then again, it's no excuse for bad software. I've only used Mongo on small sites so far, and have been loving it.
lipnitsk 16y ago
Ironically, the servers are being "upgraded" again, as we speak.
sabat 16y ago
I like the NoSQL approach as an option. But we should keep in mind: operationally, these databases/stores are comparatively new, and don't have the years and years of use that would help find and solve problems like this. It reminds me of Ebay's 3-day downtime in 1999 -- based on an Ebay mistake and an Oracle bug. Although Oracle had been around for a while in 1999, OLTP was still new, and, hence the bug. I'm not blaming a flat-out bug in this case (the cause of the severe part is still unknown?), but it could also be architectural or operator error.
- blantonl 16y ago
  I'm unable to come up with any reference to a 3 day outage regarding Oracle and Ebay in 1999. Can you provide more info on this - I'm very interested to see what happened. Edit: I found this reference to the 22hr outage that occurred, and I remember this outage, but I don't ever remember it being a 3 days outage. http://www.internetnews.com/ec-news/article.php/137251/Cost-Of-eBays-22-Hour-Outage-Put-At-2-Million.htm http://www.internetnews.com/ec-news/article.php/137251/Cost-...
  - sabat 16y ago
    I remember it being 3 days. Could be faulty memory.
  - sabat 16y ago
    Here's a link to a Forbes article about it. You're right: my memory was distorted. The outage was only 22 hours. (I'm sure it felt like three days to the Ebay admins at the time.) Fun fact: the "Steve Abatangle" quoted in the article is yours truly, and the author of the piece is Dan Lyons, now AKA Fake Steve Jobs. http://www.forbes.com/forbes/1999/0726/6402238a.html http://www.forbes.com/forbes/1999/0726/6402238a.html
- amock 16y ago
  Are you sure that was due to an Oracle bug? I heard of one eBay outage while watching a talk on ZFS that was due to misconfigured SAN volumes that overlapped and caused repeated data corruption in the Oracle DB, but wasn't related to any bug in Oracle.
smoody 16y ago
What I don't understand is this: Why do companies (like four square, twitter, etc) wait until after their first multi-hour crash before instituting a "this is how we'll communicate to users when we have downtime" process? I would assume that everyone has learned from twitter's historical mistakes at this point. I would argue that startups -- especially startups that deal with large numbers of transactions per day -- start with code and policies for communicating downtime issues first and launch the product second.
- ddlatham 16y ago
  It's not part of a minimum viable product. When you're trying to get something out the door, your downtime communication is not your top priority, so you can afford to improve it later.
  - japherwocky 16y ago
    aka "cross that bridge when you come to it"
  - marclove 16y ago
    Agreed. And they did use their Twitter accounts to communicate the problem/downtime (which even MVP's should have & use for status communications). You can be sure their status.foursquare.com blog won't have any more information than their Twitter accounts when they have unexpected downtime in the future.
- mickeyben 16y ago
  By experience, I'd say it was on their roadmap, there was even maybe a ticket about having a status.foursquare.com. But you know there's tons of tickets, tons of priorities and finally shit happens and some tasks are placed on top of the pile and become priorities.
Sukotto 16y ago
Well, that's another example for the "dangers of making untested changes to your production environment" pile. Of course, people rarely feel the need to post when everything works out... Sucks that such a popular service had such trouble. I look forward to reading any additional posts they write explaining in more detail exactly what happened.
waxman 16y ago
The site just went down again, awkwardly enough, only moments after they published their post-mortem on yesterday's outage. Clearly, as their blog post indicates, they were unable to trace the root problem. To me, the worst feeling in the world as a developer is when there's a major bug in your production site, and you can't figure out exactly why it happened. Then even after you get the site working there's that pit in your stomach of "what if it strikes again?"
- lipnitsk 16y ago
  I have a similar feeling toward my car's problems. I think it's the general issue of reproducibility that can cause such a feeling. You know the problem is there, but you're not sure of the cause, nor can you reproduce it reliably. Immediately, the issue becomes top priority because you don't know what to expect.
- jackowayed 16y ago
  Not surprisingly, it is a related problem: > UPDATE Oct. 5 8:01PM: Our server team is still working to resolve the problem. The issue is related to yesterday’s outage.
- spuz 16y ago
  After this past week, I know exactly what you mean...
dotBen 16y ago
For a service that is trying to be 'mainstream' I think their blog post is horrible. There is no way a 'regular user' is going to understand what a shard is, nor should they care. Ok, they explained sharding in laymans terms but then went on to talk about "reindexing the shard to improve memory fragmentation issues"... Woah, that means nothing to 95% of users. If you experience down time and you want your users to be sympathetic then you got to explain whats going on in terms they will understand. Sure, include a technical explanation at the bottom for those inclined, but not as part of your main body.
- marclove 16y ago
  How many "regular users" do you really think would spend the time to read about why they were down? They just know it was down and wanted it back up ASAP. And how would you better explain it to a layman? The servers crashed. We worked on it. It took longer than we wanted. They're back up again and we're working to make sure it doesn't happen again. How do you go further than that without getting technical?
  - lipnitsk 16y ago
    Actually, I think that it wasn't even all that technical. They took their time to explain what shards are, etc, so their intended audience was still power users, at best. Not saying it's good or bad, just saying how I felt when I read the post.
- harryh 16y ago
  How do you think it could have been better? We struggled a lot trying to decide how much technical detail to include. We decided that including more information (even if a lot of our users didn't understand it) was better than "something broke, it took a long time to fix." Would love to hear suggestions on this topic.
  - thegoleffect 16y ago
    Having more MongoDB-specific technical info might be a call-to-arms for other MongoDBAs who might offer their ideas/thoughts. You could also use that as a hiring/scouting opportunity. Does 4sq have an engineering blog?
haberman 16y ago
Having to manually shard your data seems like so much work when offerings like App Engine will take care of that for you. It seems like exactly the kind of thing that you shouldn't have to think about when you're trying to get a business off the ground. I can see the lock-in concerns with AppEngine, but an AppEngine level of abstraction seems so much more appropriate than manually deploying/configuring an entire infrastructure of proxies, load balancers, web servers, etc. Especially when an error can take down your whole site, like in this example.
- dmytton 16y ago
  It's not manual sharding. You specify a key a MongoDB is supposed to handle everything for you. There are manual operations you can perform if you want (like moving data or splitting) but normally you'd let Mongo handle it all.
- cyu 16y ago
  I can't imagine any co with that much investment money betting on a platform that they can never take ownership of.
trustfundbaby 16y ago
"but... MongoDB is web scale." --- commenter
- storm 16y ago
  Indeed. http://www.xtranormal.com/watch/6995033 http://www.xtranormal.com/watch/6995033
bhiggins 16y ago
If there's anyone from Foursquare here, I'm interested in what monitoring you have in place. How long had the shard been poorly performing before you noticed? Do you use anything to monitor MongoDB in particular, or load on servers, anything like that?
dmytton 16y ago
This posting in the MongoDB mailing list provides more detail from the developers: http://groups.google.com/group/mongodb-user/browse_thread/thread/66752f49af68619 http://groups.google.com/group/mongodb-user/browse_thread/th...
- dowskitest 16y ago
  Sounds like a side effect of relying on MMAP (and not doing compaction). "Basically, the issue is that if data migrates to a new shard, there is no re-compaction yet in the old shard of the old collection. So there could be small empty spots throughout it which were migrated out, and if the objects are small, there is no effective improvement in RAM caching immediately after the migration." - Dwight Merriman (at the link in the parent). "The kernel is able to swap/load 4k pages. For a page to be idle from the point of view of the kernel and its LRU algorithm, what is needed is that there are no memory accesses in the whole page for some time." -antirez from http://antirez.com/post/what-is-wrong-with-2006-programming.html http://antirez.com/post/what-is-wrong-with-2006-programming....
cagenut 16y ago
I notice foursquare.com resolves to ec2, was the mongodb that got overloaded also on ec2? Can you tell us in what way did it get "overloaded" (iowait, mem/swap, raw-cpu)?
ankimal 16y ago
Firstly, as a free user you really shouldn't be asking for HA. (I m assuming their paid customer are kicking some butt as I speak .. or maybe not .. the site is back up :)). However, as a business you really want to give ALL your customers HA. Its not just a reputation thing, its a "we love you all equally" kinda attitude. As for MongoDB, we ve been using in production for small insignificant things. FWIW, they have replication http://www.mongodb.org/display/DOCS/Replication http://www.mongodb.org/display/DOCS/Replication and some cool new features like Replica Sets for failover and redundancy. Maybe they missed a trick? I think the apology post was totally fair and he did categorically mention ".. This blog post is a bit technical. It has the details of what happened, and what we’re doing to make sure it doesn’t happen again in the future.". They could have dilly-dallied with words and said "we had a technical failure of a data nature" and that would have been just been plain stupid. So thanks for the detailed technical write up and hope there is more to follow.
- dhimes 16y ago
  HA?
  - allwein 16y ago
    High Availability
    - dhimes 16y ago
      Thanks
jscore 16y ago
I'm so happy they're back up, here I was thinking the world would come to a screeching halt when people cannot check-in to places. Seriously though, are they THAT important?
- JabavuAdams 16y ago
  No, they're not, but they have to dance the dance and say / do the right things if they want to keep their users and investors happy.

189 points by abraham 16y ago

we noticed that one of these shards was performing poorly because a disproportionate share of check-ins were being written to it. I'd love to know the root cause behind this specific issue. Was this a behavioral issue within the user base, or a technical problem that routed check-ins to this specific shard more than others? Since they mentioned they partition their shards by userId that would probably rule out their routing process. I wonder if there was some event that caused a certain sharded subsection of users to start sending so many checkins? And since this was a subset of userIds on the same shard - could this have been a targeted DOS or SPAM event? I'm making a very conscious migration to MongoDB so I'm very interested to hear what the root cause of this was.

brown9-2 16y ago

Interestingly the cause of the most severe part of their downtime seems to still be unknown to them: As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one. We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go down. To those who use MongoDB - does this sound like something that might have been caused by MongoDB itself, or Foursquare's use of it?

houseabsolute 16y ago
I'm also wondering why this kind of intervention is necessary at all. The NoSQL solution we use at work has load based automatic splitting, and I'd have thought (though I haven't confirmed) that this would be an obvious feature to include.
- icey 16y ago
  Which NoSQL store do you use?
  - houseabsolute 16y ago
    http://labs.google.com/papers/bigtable.html http://labs.google.com/papers/bigtable.html
    - staunch 16y ago
      That looks potentially useful in some cases. Can you send me the code/docs for that?
      - ddlatham 16y ago
        http://hbase.apache.org/ http://hbase.apache.org/
        ifesdjeen 16y ago
        Does it work that well for production for you? Really, that's extremely interesting.
      - rb2k_ 16y ago
        Also look at dynamo based systems like cassandra and riak (riak seems to have a better load balancing at the moment, cassandra is a bit more "bumpy")
- dmytton 16y ago
  Load based splitting is on the MongoDB roadmap, but doesn't exist yet.
- fizx 16y ago
  Just because the feature exists, doesn't mean it works.
- mcfunley 16y ago
  I would speculate that it's a poorly-chosen shard key. MongoDB's built-in sharding uses range-based indexing. If you choose user_id as your shard key, and those are autoincrementing integers, then you're screwed if newer users tend to be more active on average than older ones.
  - jrockway 16y ago
    Wait, people shard on a key other than something approximately random, like an sha1 hash!?
    - mcfunley 16y ago
      Where I work (Etsy) we keep an index server that maps each user to a shard on an individual basis. There are a number of advantages to it. For example, if one user generated a ton of activity they could in theory be moved to their own server. Approximately random works for the initial assignment. Flickr works the same way (not by coincidence, since we have several former Flickr engineers on staff).
      - jrockway 16y ago
        Sounds like a good system. I've noticed that people tend to do things like shard based on even/odd, and then they realize that they need three databases. I've never had either problem though... but if I ever need to shard I plan on doing it based on object ID. Then one request can be handled by multiple databases, "for free", increasing both throughput and response time.
        fizx 16y ago
        Even/odd isn't the end of the world, but you would then be best jumping to mod 4.
        moe 16y ago
        Actually for anonymous sharding (without a central index) a consistent hash is about the closest you can get to ideal distribution and flexibility. I haven't looked but I presume that's what mongo uses under the hood for their auto-sharding, too.
dmytton 16y ago
Without knowing more about their schema, it's hard to speculate. However, let's assume they had a single, large "checkins" collection, sharded by user ID as they said. That would be distributed across the shards. If the mongod process is down on one shard and therefore the shard isn't responding, the data just wouldn't be available. But it sounds like the shard was still up. As per the sharding FAQ[1]: "What if a shard is down or slow and I do a query? If a shard is down, the query will return an error. If a shard is responding slowly, mongos will wait for it. You won't get partial results." They said they had a performance issue on the overloaded shard so perhaps it's possible that mongo believed the shard to be up, when it was instead overloaded. This meant any queries just waited, taking down the whole site. As I said, this is speculation and as a massive production user of MongoDB ourselves[2], I'm interested to know more. [1] http://www.mongodb.org/display/DOCS/Sharding+FAQ http://www.mongodb.org/display/DOCS/Sharding+FAQ [2] http://blog.boxedice.com/2010/02/28/notes-from-a-production-mongodb-deployment/ http://blog.boxedice.com/2010/02/28/notes-from-a-production-...
- marclove 16y ago
  But it wasn't the overloaded shard that brought the site down (it was only slowing things down), it was the addition of the new shard: "For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go down."
  - dmytton 16y ago
    If the data was being moved it could cause even more load on an already overloaded shard.
- VladRussian 16y ago
  "What if a shard is down or slow and I do a query? If a shard is down, the query will return an error. If a shard is responding slowly, mongos will wait for it. You won't get partial results." an approach to CAP that chooses consistency over availability. I wonder why as it isn't a bank and they could have chosen the availability instead.
scg 16y ago
At 6:30pm EST, we determined the most effective course of action was to re-index the shard, which would address the memory fragmentation and usage issues. Sounds similar to some of the problems we're facing with MongoDB too. Indexes fragment a lot, especially if you delete documents frequently. You have to reindex everything eventually. MongoDB offers background indexing (except for _id fields!) but it's not really an option if you need the indexes at all times to keep executing queries. Also, you'll need to reindex _id eventually and there's currently no way to do that without blocking the server.
rb2k_ 16y ago
> To those who use MongoDB - does this sound like something that might have been caused by MongoDB itself, or Foursquare's use of it? MongoDB still has problems with locking too much for certain operations. The 10gen guys keep on introducing "yields" to those operations, but the simple (and fast) database/process-level locking could very well cause this... (not a 100% 'yes', but it seems possible)

marclove 16y ago

Well I have nothing constructive to add, but it certainly makes me rethink using MongoDB in any sort of serious production environment. I've been hesitant to use MongoDB for this very reason...what happens when things go wrong? I'm certainly not an expert in how to handle those situations, and there aren't very many of those experts out there. Unfortunately for Foursquare, they got caught with their pants down, not having someone on staff who really understands all the ins & outs of a database technology their entire service depends upon.

dmytton 16y ago
You can get commercial support from 10gen (MongoDB developers) as we do for my company. It's just like MySQL providing support for their enterprise DB. Of course MySQL is much more widely used than MongoDB so there is more community help and knowledge.
- marclove 16y ago
  Right, I'm aware. Not everybody can afford a contract though, nor should they have to in order to avoid major outages like this. But I would like to know if Foursquare has a commercial support contract with 10Gen. If they didn't, why not? especially for a service that big? If they did, how was it that 10Gen took that long to fix the problem?
  - sachinag 16y ago
    They share USV as an investor. I'd be astonished if they didn't.
    - marclove 16y ago
      Heh...that's one hell of a commercial support contract then. Should be interesting to read 10gen's own detailed post-mortem.
  - steveklabnik 16y ago
    I'm sure that if you're the size of Foursquare, you can afford the contract. Then again, it's no excuse for bad software. I've only used Mongo on small sites so far, and have been loving it.

lipnitsk 16y ago

Ironically, the servers are being "upgraded" again, as we speak.

sabat 16y ago

I like the NoSQL approach as an option. But we should keep in mind: operationally, these databases/stores are comparatively new, and don't have the years and years of use that would help find and solve problems like this. It reminds me of Ebay's 3-day downtime in 1999 -- based on an Ebay mistake and an Oracle bug. Although Oracle had been around for a while in 1999, OLTP was still new, and, hence the bug. I'm not blaming a flat-out bug in this case (the cause of the severe part is still unknown?), but it could also be architectural or operator error.

blantonl 16y ago
I'm unable to come up with any reference to a 3 day outage regarding Oracle and Ebay in 1999. Can you provide more info on this - I'm very interested to see what happened. Edit: I found this reference to the 22hr outage that occurred, and I remember this outage, but I don't ever remember it being a 3 days outage. http://www.internetnews.com/ec-news/article.php/137251/Cost-Of-eBays-22-Hour-Outage-Put-At-2-Million.htm http://www.internetnews.com/ec-news/article.php/137251/Cost-...
- sabat 16y ago
  I remember it being 3 days. Could be faulty memory.
- sabat 16y ago
  Here's a link to a Forbes article about it. You're right: my memory was distorted. The outage was only 22 hours. (I'm sure it felt like three days to the Ebay admins at the time.) Fun fact: the "Steve Abatangle" quoted in the article is yours truly, and the author of the piece is Dan Lyons, now AKA Fake Steve Jobs. http://www.forbes.com/forbes/1999/0726/6402238a.html http://www.forbes.com/forbes/1999/0726/6402238a.html
amock 16y ago
Are you sure that was due to an Oracle bug? I heard of one eBay outage while watching a talk on ZFS that was due to misconfigured SAN volumes that overlapped and caused repeated data corruption in the Oracle DB, but wasn't related to any bug in Oracle.

smoody 16y ago

What I don't understand is this: Why do companies (like four square, twitter, etc) wait until after their first multi-hour crash before instituting a "this is how we'll communicate to users when we have downtime" process? I would assume that everyone has learned from twitter's historical mistakes at this point. I would argue that startups -- especially startups that deal with large numbers of transactions per day -- start with code and policies for communicating downtime issues first and launch the product second.

ddlatham 16y ago
It's not part of a minimum viable product. When you're trying to get something out the door, your downtime communication is not your top priority, so you can afford to improve it later.
- japherwocky 16y ago
  aka "cross that bridge when you come to it"
- marclove 16y ago
  Agreed. And they did use their Twitter accounts to communicate the problem/downtime (which even MVP's should have & use for status communications). You can be sure their status.foursquare.com blog won't have any more information than their Twitter accounts when they have unexpected downtime in the future.
mickeyben 16y ago
By experience, I'd say it was on their roadmap, there was even maybe a ticket about having a status.foursquare.com. But you know there's tons of tickets, tons of priorities and finally shit happens and some tasks are placed on top of the pile and become priorities.

Sukotto 16y ago

Well, that's another example for the "dangers of making untested changes to your production environment" pile. Of course, people rarely feel the need to post when everything works out... Sucks that such a popular service had such trouble. I look forward to reading any additional posts they write explaining in more detail exactly what happened.

waxman 16y ago

The site just went down again, awkwardly enough, only moments after they published their post-mortem on yesterday's outage. Clearly, as their blog post indicates, they were unable to trace the root problem. To me, the worst feeling in the world as a developer is when there's a major bug in your production site, and you can't figure out exactly why it happened. Then even after you get the site working there's that pit in your stomach of "what if it strikes again?"

lipnitsk 16y ago
I have a similar feeling toward my car's problems. I think it's the general issue of reproducibility that can cause such a feeling. You know the problem is there, but you're not sure of the cause, nor can you reproduce it reliably. Immediately, the issue becomes top priority because you don't know what to expect.
jackowayed 16y ago
Not surprisingly, it is a related problem: > UPDATE Oct. 5 8:01PM: Our server team is still working to resolve the problem. The issue is related to yesterday’s outage.
spuz 16y ago
After this past week, I know exactly what you mean...

dotBen 16y ago

For a service that is trying to be 'mainstream' I think their blog post is horrible. There is no way a 'regular user' is going to understand what a shard is, nor should they care. Ok, they explained sharding in laymans terms but then went on to talk about "reindexing the shard to improve memory fragmentation issues"... Woah, that means nothing to 95% of users. If you experience down time and you want your users to be sympathetic then you got to explain whats going on in terms they will understand. Sure, include a technical explanation at the bottom for those inclined, but not as part of your main body.

marclove 16y ago
How many "regular users" do you really think would spend the time to read about why they were down? They just know it was down and wanted it back up ASAP. And how would you better explain it to a layman? The servers crashed. We worked on it. It took longer than we wanted. They're back up again and we're working to make sure it doesn't happen again. How do you go further than that without getting technical?
- lipnitsk 16y ago
  Actually, I think that it wasn't even all that technical. They took their time to explain what shards are, etc, so their intended audience was still power users, at best. Not saying it's good or bad, just saying how I felt when I read the post.
harryh 16y ago
How do you think it could have been better? We struggled a lot trying to decide how much technical detail to include. We decided that including more information (even if a lot of our users didn't understand it) was better than "something broke, it took a long time to fix." Would love to hear suggestions on this topic.
- thegoleffect 16y ago
  Having more MongoDB-specific technical info might be a call-to-arms for other MongoDBAs who might offer their ideas/thoughts. You could also use that as a hiring/scouting opportunity. Does 4sq have an engineering blog?

haberman 16y ago

Having to manually shard your data seems like so much work when offerings like App Engine will take care of that for you. It seems like exactly the kind of thing that you shouldn't have to think about when you're trying to get a business off the ground. I can see the lock-in concerns with AppEngine, but an AppEngine level of abstraction seems so much more appropriate than manually deploying/configuring an entire infrastructure of proxies, load balancers, web servers, etc. Especially when an error can take down your whole site, like in this example.

dmytton 16y ago
It's not manual sharding. You specify a key a MongoDB is supposed to handle everything for you. There are manual operations you can perform if you want (like moving data or splitting) but normally you'd let Mongo handle it all.
cyu 16y ago
I can't imagine any co with that much investment money betting on a platform that they can never take ownership of.

trustfundbaby 16y ago

"but... MongoDB is web scale." --- commenter

storm 16y ago
Indeed. http://www.xtranormal.com/watch/6995033 http://www.xtranormal.com/watch/6995033

bhiggins 16y ago

If there's anyone from Foursquare here, I'm interested in what monitoring you have in place. How long had the shard been poorly performing before you noticed? Do you use anything to monitor MongoDB in particular, or load on servers, anything like that?

dmytton 16y ago

This posting in the MongoDB mailing list provides more detail from the developers: http://groups.google.com/group/mongodb-user/browse_thread/thread/66752f49af68619 http://groups.google.com/group/mongodb-user/browse_thread/th...

dowskitest 16y ago
Sounds like a side effect of relying on MMAP (and not doing compaction). "Basically, the issue is that if data migrates to a new shard, there is no re-compaction yet in the old shard of the old collection. So there could be small empty spots throughout it which were migrated out, and if the objects are small, there is no effective improvement in RAM caching immediately after the migration." - Dwight Merriman (at the link in the parent). "The kernel is able to swap/load 4k pages. For a page to be idle from the point of view of the kernel and its LRU algorithm, what is needed is that there are no memory accesses in the whole page for some time." -antirez from http://antirez.com/post/what-is-wrong-with-2006-programming.html http://antirez.com/post/what-is-wrong-with-2006-programming....

cagenut 16y ago

I notice foursquare.com resolves to ec2, was the mongodb that got overloaded also on ec2? Can you tell us in what way did it get "overloaded" (iowait, mem/swap, raw-cpu)?

ankimal 16y ago

Firstly, as a free user you really shouldn't be asking for HA. (I m assuming their paid customer are kicking some butt as I speak .. or maybe not .. the site is back up :)). However, as a business you really want to give ALL your customers HA. Its not just a reputation thing, its a "we love you all equally" kinda attitude. As for MongoDB, we ve been using in production for small insignificant things. FWIW, they have replication http://www.mongodb.org/display/DOCS/Replication http://www.mongodb.org/display/DOCS/Replication and some cool new features like Replica Sets for failover and redundancy. Maybe they missed a trick? I think the apology post was totally fair and he did categorically mention ".. This blog post is a bit technical. It has the details of what happened, and what we’re doing to make sure it doesn’t happen again in the future.". They could have dilly-dallied with words and said "we had a technical failure of a data nature" and that would have been just been plain stupid. So thanks for the detailed technical write up and hope there is more to follow.

dhimes 16y ago
HA?
- allwein 16y ago
  High Availability
  - dhimes 16y ago
    Thanks

jscore 16y ago

I'm so happy they're back up, here I was thinking the world would come to a screeching halt when people cannot check-in to places. Seriously though, are they THAT important?

JabavuAdams 16y ago
No, they're not, but they have to dance the dance and say / do the right things if they want to keep their users and investors happy.