9 ms·
MongoDB's lead developer: Foursquare outage post mortem
- chrislloyd 16y ago117 GB of data and 300 B for a checkin means that Foursquare has had 408,944,640 checkins.
- jranck 16y agoFor example, if we had notifications in place to alert us 12 hours earlier that we needed more capacity, we could have added a third shard, migrated data, and then compacted the slaves. Where did Foursquare find their engineers? I hope no one lost their job here but this is pretty elementary stuff.
- driverdan 16y agoA bit harsh but a good point. There are a few red flags here. Resource monitoring on the servers, like you mentioned, seems pretty obvious. Especially considering that the system was almost guaranteed to crash when it ran out of RAM. Sharding needs to be managed in a way that evenly distributes the data. I would never do it by user ID unless a proper analysis of the data showed it to be a fairly even distribution.
- slantyyz 16y agoIt's probably a technical debt thing. FourSquare is immensely popular, and the team probably meant to have those checks and balances in place but got 'too busy' to implement them. That doesn't excuse anything, but these oversights can happen even when you've got primo talent on board.
- deleted 16y ago[deleted]
- harryh 16y agoIt's true that this is elementary in and of itself, but looking at things with a bit of a wider lens shows the complexity. We're a small engineering team (10 people) working on a product that is growing extremely fast both in terms of usage and feature set. Meanwhile we're also pretty much constantly re-architecting things to keep up with growth and also doing the immense work of growing the company up from 3 people to 33 and beyond (this has turned out to be WAY HARDER than I would have guessed going in). Further, there are lots and lots of different things that we need to be monitoring at any different time to make sure that everything is going ok and we aren't about to run into a wall. Automated tools can help a lot with this, but these tools still need to be properly set up and maintained. I'm not saying we didn't screw up. We had 17 hours of downtime over two days. We screwed up bad, and we feel horrible about it, and are doing a lot to make sure that we don't screw up the same way again. But it's not because we're morons that never thought about the fact that we should be monitoring memory usage. We just got overwhelmed with the complexity of all that we're doing at once. -harryh, foursquare eng lead
- chewbranca 16y agoI agree completely about the complexity of monitoring solutions for small teams with fluctuating applications. However, something as simple as htop running on an extra monitor would have alerted you of this issue long before downtime resulted.
- cullenking 16y agoAwww come on, it's easy to say from an outside perspective. Regardless, the problem was handled well in the end, and we all get the benefit of understanding these limitations better. I think us tech people got the good side (information) out of this ordeal :)
- chewbranca 16y agoI work on a team of 2 where I'm responsible for a handful of servers. I chimed in because I'm in a similar position, I've been looking for a monitoring solution for a while now. Things like nagios and zenoss are over kill, but lack of time has prevented me from finding an ideal solution. That said, I keep htop open and running at all times, and its saved my ass on more than one occasion. I say htop because of the color coding it provides, if things start going red it attracts my attention.
- jasonwatkinspdx 16y agoWhen you're a small company growing fast you have an endless list of things you really should have done a long time ago. That's no excuse for not preventing a predictable and painful failure, but do try to remember it's a lot easier to criticize as an observer with no time commitments.
- tripngroove 16y agoDisclaimer: I work at Cloudkick. We can help you all with these problems. Here's how fast/easy it is: 1. Create an account (~30 sec) 2. Add your cloud credentials (~45 sec) 3. Install the monitoring agent (~90 sec/node) 4. Create a CPU/Memory/Disk monitor, query-targeted all your servers (~60 sec) (example: "provider:EC2") 5. You get an email whenever the monitors you created reach the thresholds you set There are a lot of other cool features - check 'em out on the site: www.cloudkick.com All our plans are free for 30 days.
- daleharvey 16y agoWhile I usually frown on such blatant self promotion, I couldnt help but upvote this for being such a good advertisement, I want to use you now.
- al_james 16y agoOr you could install Ganglia for free.
- dedward 16y agoPerhaps it's just my own background that pushes me this way - but as soon as my deployment was growing ot eat up anything over about 60% of my available memory, I'd be looking at deploying more. Your high water mark for spinning up more nodes and/or upgrading shouldn't be your maximum physical ram capacity - that's just nuts.
- BenSchaechter 16y agoThat was one of the most well-written, to-the-point explanations I have read in a long time.
- famousactress 16y agoI wonder if there ought to be a move to allow for sharded systems to lump groups/pages nicely.. kind of like MySQL's partitioning.. where you won't run into the issue of sparsing out pages when you migrate data.. instead you'd effectively migrate pages.
- ehwizard 16y agoYes - something that keeps data organized my shard key is being planned.
- narrator 16y agoWell Foursquare's Scala/Lift front-end sure has been holding up nicely despite all the FUD about Lift's stateful architecture. Funny that Mongo DB, which gets all the scalability hype, is the first thing to have trouble scaling.
- suhail 16y agoMySQL has trouble scaling if you don't understand how to use it or monitor it. Not sure I agree with the last part of your comment at all.
- deleted 16y ago[deleted]
- rit 16y agoThe described problem - imbalance in shard allocation - is not one that's specific to MongoDB. In fact one of the huge choices one has to make when deploying say, Cassandra, is the partitioning algorithm. Sequential partitioning (saying say, users a-l go on Partition 1 and m-z go on #2) gives you certain capabilities for range queries but also risks you overloading if one particular user is significantly larger than others. Cassandra recommends random partitioning as a way of better balancing your data across shards. You're going to run into the same problem on just about any sharded setup (With any software) - how do you make sure that you are distributing new chunks/documents/rows in a way that doesn't overload any given server.
- tav 16y agoOr your datastore can take care of doing that for you, c.f. BigTable. This is one of my main gripes with most of the current set of NoSQL offerings — they leave too many decisions in the hands of developers. Whilst it would definitely be advantageous for all developers to understand the intricacies of various CPU and OS scheduling algorithms, it's not an issue that most developers have to deal with directly. The App Engine datastore, in particular, proves that it is possible to create NoSQL datastores which don't force developers to think about issues like load balancing.
- zaidf 16y agoIs it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?
- suhail 16y agoIt's okay to have hundreds of terrabytes of data but you need to keep your "working set" hot. Even if you have 10TB of data, 1TB of it might actually be hot. Monitoring your working set is tough.
- ericflo 16y agoYea, but that doesn't look like what they're describing here. They seem to be describing having the entire dataset in memory.
- btilly 16y agoIt is easy to turn swap off. There are many reasons why it can be good to turn swap off in servers. (But only if you've got sufficient monitoring to hear about it before you run out of RAM.)
- btilly 16y agoDepends on what you are doing. If you have high volume and care about latency, you pretty much have to put everything in RAM. If your disk is spinning at 6000 rpm, a given sector spins past 100 times per second, which makes lookup time anywhere from 0 to 1/100th of a second, or 1/200th of a second on average. Disks only let you look for a limited number of things at once, so you can do less than 1000 disk seeks per second. Period. If data is living on disk and you have high query volume, it is really, really easy to blow past that limit. The solution is to shard data on multiple machines in RAM. This gives a fixed cost per unit of RAM. As long as you don't exceed available RAM, it works well. Luckily it isn't hard to monitor available RAM and respond in advance. (They didn't do that in this case.) If you don't care about latency, or have a lower query volume, then you can live with data on disk, and frequently accessed data in RAM. A few well-designed caching layers in front can give you even more headroom. This is much cheaper. But has more complicated failure modes, and they can be tricky to monitor properly.
- SemanticFog 16y agoI like the way MongoDB describes the flaws in their own system, but places the blame (four) squarely on the customer, where it belongs. It wasn't a random failure or a sudden spike that caused the crash -- it was completely predictable growth. Foursquare had already experienced the problem once, and they had solved it. All they needed to do was monitor their growth and iterate that solution. Sure, Foursquare could have had a better sharding algorithm, but that would only have put off the crash a bit longer. This is a very basic failure -- not monitoring a system that you know is steadily growing.
- SriniK 16y agoI think they mongodb guys are correct - it is an issue with app architecture and app monitoring failure.
- redthrowaway 16y agoThe main thing I took from this incident is how much being open and honest about issues improves a company's image. Foursquare and 10gen could have very easily played the blame game, or kept their cards close to their chests, and both would have come off poorly. Instead, they described the problem, owned up to their role in it, and laid out a framework for how to avoid the problem in the future. After reading about the issue, I come away with respect for both groups. Sure, mistakes were made, but they treated their users like adults and took responsibility for what they did wrong. Good job, guys. If only the likes of Apple followed this example.
- slantyyz 16y agoForget about Apple, it would have been nice if BP followed this example.
- SkyMarshal 16y agoProblem is lawsuits. Admitting guilt can and is used against you in court. Hence, US businesses don't apologize for anything. This is a refreshing exception.
- ceejayoz 16y agoIf there was anything Foursquare could be sued over here, we'd be seeing a different story.
- SkyMarshal 16y agoTrue.
- megablast 16y agoI see it as a very American characteristic, from the people I have worked with before. Many Americans will never admit fault, never admit they do not know something, and always looking for someone else to blame. I think this comes from the culture of large companies. I have not seen this at smaller US companies, and other companies I have worked at. Of course, not all Americans, but I was surprised when I first saw this.
- michaelhalligan 16y agoNet impact analysis of outage: A bunch of socially awkward technophiles couldn't tell their friends that they were eating lunch.
- slantyyz 16y agoI wonder if burglaries also went down during the outage, since the bad guys couldn't tell if their targets were at home or busy being the Mayor at the local bowling alley.
- moondowner 16y agoOne thing that I admire about Foursquare is that they don't only use the technology (MongoDB, Lift Framework, etc..) but also invest in it. If it wasn't for them I think that Lift wouldn't be such an advanced framework as it is now.
- harryh 16y agoThanks! We do what we can (we're very busy these days!), and only hope to be able to do more of this sort of thing in the future.
- voxxit 16y agoWhy they have only two database servers running (with their database in memory, no less) with 200 million check-ins, is completely beyond me.
- seiji 16y agoThey are dealing with web scale sharded NoSQL realtime geo scala. Old rules don't apply when 80% of the words describing your company didn't exist two years ago.
- doty 16y agoI honestly cannot tell whether or not that comment was serious.
- epochwolf 16y agoAny comment with the words "web scale" should be considered a joke until proven otherwise. :)
- look_lookatme 16y agoI cracked up at this. Well played.
- bl4k 16y agoJust as I was thinking that this thread was getting worse the further I scroll down I come across the best summation of this story
- mrkurt 16y agoThey actually said 4. Two shards with slaves for redundancy. It was apparently working fine until they lost track of what was going on. :)
- marclove 16y agoYea but like you said, since half of them are just for redundancy, they were in effect only running 2 database servers. I must say I find it stunning that a service with that much traffic, especially one with an infrastructure that requires the entire database to be in-memory, would be operating with just 2 database servers running on EC2. I know hindsight is 20/20, but I can't imagine the foresight was any worse than 20/30.
- donaldc 16y agoIt sounds like even if the check-ins had grown evenly across the two shards, that would have only saved them for about another month. Without close monitoring of the growth in memory usage, foursquare was still due for an outage.
- harryh 16y agoThis is 100% true. We were always planning on moving to more shards, we just thought we had more time to do it than we did.
- donaldc 16y agoIn essence, although we had moved 5% of the data from shard0 to the new third shard, the data files, in their fragmented state, still needed the same amount of RAM. This can be explained by the fact that Foursquare check-in documents are small (around 300 bytes each), so many of them can fit on a 4KB page. Removing 5% of these just made each page a little more sparse, rather than removing pages altogether. Interestingly, this is one of the reasons antirez gives as to why redis will not be using the built-in OS paging system, but instead will use one custom-written for redis' needs.
- IgorPartola 16y agoI couldn't help but think of that exact issue. I suppose once compacting the data online is built in, this particular issue won't come up again. At the same time, when a machine is overloaded, often times you have even bigger problems. For example, if you are out of memory, you may not be able to create another SSH process to get at the box.
- kunley 16y agoThe situation antirez had in mind can be remedied by occasionally using a tool like vmtouch to steer what's in the OS cache.
- IgorPartola 16y agoWell, the other big advantage being that the OS has no idea about the particular kinds of data you want to store, whereas Redis has quite a bit more data to work with. But yes, vmtouch looks useful for these types of situations (thanks).
- deleted 16y ago[deleted]
- Nate75Sanders 16y agoWe need an "Ask HN" for cool tools. Maybe there's already a SO question about it. I hadn't heard of vmtouch and probably tons of other things people here use.
- jasonwatkinspdx 16y agoBuilding sharded systems isn't as simple as throwing consistent hashing into the code and calling it a day. You have to think carefully about what happens when nodes exceed capacity. There is good work in academia on distribution algorithms that gracefully handle reaching capacity (along with data center structure such as rack awareness) [1]. Alternately, if your algorithm doesn't handle a shard reaching capacity you need to have the monitoring and processes in place to ensure you always add more capacity and rebalance before running out. Also, this validates Redis's VM position about 4KB pages being to large to properly manage data swapping in web application storage. [1]: http://www.ssrc.ucsc.edu/Papers/weil-sc06.pdf http://www.ssrc.ucsc.edu/Papers/weil-sc06.pdf
- t_crayford 16y agoI'm wondering if the decision to shard based on users was taken with data (ie, at that point did each user have a roughly similar number of check-ins); if not, hashing by that seems kind of fail for that kind of app.
- rcoder 16y agoFrom the email thread, it sounds like the decision to shard on UID was made mostly to increase locality of data, so that you didn't have to query more than one node to get a single user's data. There's no silver bullet here. Hashing on insertion order would basically guarantee that writes would favor one node over another, which random hashes would force you to aggregate results from all available nodes for each query.
- jasonwatkinspdx 16y agoThis stuff can be very counter intuitive. Locality may not be what you want. For example, last I heard google's search index was sharded by document rather than by term. That sounds odd, since if it was sharded by term, then a given search would only need to go to a handful of servers (one for each term) and then the intermediate result combined. But with it sharded by document, every query has to go to all the nodes in each replica/cluster. It ends up that's not as bad as it seems. Since everything is in ram, they can answer "no matches" on a given node extremely quickly (using bloom filters mostly in on processor cache or the like). They also only send back truly matching results, rather than intermediates that might be discarded later, saving cluster bandwidth. Lastly it means their search processing is independent of the cluster communication, giving them a lot of flexibility to tweak the code without structural changes to their network, etc. Does that mean everyone doing search should shard the same way? Probably not. You have to design this stuff carefully and mind the details. Using any given data store is not a silver bullet.
- justlearning 16y agofor those who don't want to login to see the post: http://gist.github.com/616108 http://gist.github.com/616108
- alexpopescu 16y agoWhile the details are very interesting, there are still many questions to be answered (on both sides): - how difficult would be to bring up read-only replicas? (hopefully that should take much less than 11 hours + 6hours) - why the 3rd shard could accommodate only 5% of the data? - how can you plan capacity when using the "wrong" sharding? (basically leading to unpredictable distributions) I have posted the rest of the questions here: http://nosql.mypopescu.com/post/1265191137/foursquare-mongodb-outage-post-mortem http://nosql.mypopescu.com/post/1265191137/foursquare-mongod... as I hope to get some more answers.
- harryh 16y ago> how difficult would be to bring up read-only replicas? > (hopefully that should take much less than 11 hours > 6hours) Bringing up read only replicas would have been easy, but our appservers are not currently designed to read data from multiple replicas so it wouldn't have helped. We hope to make architectural changes to allow for this sort of thing in the future but aren't there yet. > - why the 3rd shard could accommodate only 5% of the data? The issue wasn't the amount of data the 3rd shard could accomodate, but the rate at which data could be transfered off the 1st (overloaded) shard onto the 3rd shard. > how can you plan capacity when using the "wrong" > sharding? (basically leading to unpredictable > distributions) We weren't really using the "wrong" sharding. And even the uneven distribution we saw (about 60%/40%) wasn't totally horrible. Not 100% understanding your question here.
- alexpopescu 16y agoI didn't mean "wrong" in the full sense of the word. What I actually mean is that sharding based on user id will most probably not give/guarantee predictable distributions of the writes generated by users (basically there can be no guarantee that chunk#1 of users will always generate the aprox. same amount of data as chunk#2 of users).
- dirkgadsden 16y agoWhat exactly is the rate like for transferring data across MongoDB instances on EBS? Was the overhead of going across EBS volumes the major factor in the length of the downtime? The 60%/40% sharding distribution doesn't seem too bad in the big picture; are there any plans to be evening that out?
- swaits 16y agoRight, kick ass. Well, don't want to sound like a dick or nothin', but, ah... it says on your chart that you're fucked up. Ah, you talk like a fag, and your shit's all retarded. What I'd do, is just like... like... you know, like, you know what I mean, like...
- swaits 16y agoGeez. No sense of humor. It's a line from Idiocracy. Hilarious.
- tmountain 16y agoThe "give a crap" factor at 10gen is nothing short of amazing. The company I work at has been using Mongo for a while, and anytime we've had an issue, Eliot has been right there to help us. Mongo is a great product, but like anything, it has its limits. Learning those limits and taking the time to plan your infrastructure is a mandatory part of adopting any technology, and Mongo is no exception. On a more technical note, it would be nice to have a way to compact indexes online without having to resort to doing so on a slave, but Mongo is a minimum two server product to begin with, so it's not the end of the world. Overall it's a great datastore, and it's only going to get better in the next few years.
- cies 16y ago> Mongo is a minimum two server product to begin with are you sure? i thought it runs well on one server and even in some sort of run-locally setup.
- tmountain 16y agoYes, if you care about your data, it's a two server product. Single server redundancy isn't being added until 1.8. Until then, if the server terminates abnormally, there's potential for data corruption. The solution right now is to run a slave. http://www.mongodb.org/display/DOCS/Durability+and+Repair http://www.mongodb.org/display/DOCS/Durability+and+Repair
- sausagefeet 16y agoThis solution strikes me as best-case-scenario error. What happens if the connection between my slave and master goes down then my master is unplugged? What happens if both slave and master are unplugged, can both databases be corrupted? What am I missing?
- tmountain 16y agoIf the connection goes away and your master is unplugged, you will need to run a repair on the master and then bring the slave back into sync. If both are unplugged, you will have to run the repair on both. This obviously isn't ideal, and that's probably why single server durability is the biggest priority for the next release.
- 2mur 16y agoI don't use NoSQL (yet) but I'm curious if there would be a better key to shard by (instead of user_id)?
- jmulho 16y agoWhy on earth would you want to keep 236 million check-in documents (66 gig / 300 bytes) in memory ? Here is an idea: write an algorithm that keeps the most recently used 100 million check-in documents in memory. That'll save 38 gig of RAM. Or how about this idea from the 1970s: write an algorithm that keeps the most recently used 4 gig of check-in documents in memory. That will save 62 gig of RAM, and the most recently active 14 million users (4 gig / 300 bytes) will still enjoy instantaneous response. Can someone help me out with the math here?
- ww520 16y agoIsn't that what the on-demanding disk cache (and paging) from the OS give you? Only the data being accessed and used are in memory. Unless they're constantly doing data churning over the whole dataset, there is no need to keep everything in memory.
- silentbicycle 16y agoPaging doesn't happen in 300 byte chunks, though. It's usually 4 KB, so if a record or one of twelve neighbors has been touched, it stays in. The granularity isn't right.
- ww520 16y agoThe problem isn't the page granularity, which is just a convenient size for the general case and works pretty well in most cases. The problem is to not keep excessive data in memory; in this case, they are keeping ALL data in memory due to some architectural decisions. OS caching and paging have work pretty well in keeping the frequently used data in memory and leave the rest on disk, and thus utilize the system resource more efficiently.
- jmulho 16y agoI am talking about the database looking in memory for what it needs to answer a request; if it doesn't find what it needs, it reads from disk. Since it has been allowed limited memory, when it put something new in memory, it has to kick something old out. It uses a simple algorithm to identify what to keep and what to kick out. One of the benefits of this (obvious) design is that when too many users are active at the same time to keep all of them in memory, a few of them experience a small lag (as opposed to all of them experiencing a complete crash). If you have a few terabytes of disk then you have to really be asleep at the wheel (like for a decade) to totally crash.
- invertedlambda 16y agoMongoDB is web scale.
- invertedlambda 16y agoYou guys are lame - you obviously haven't seen: http://www.xtranormal.com/watch/6995033/ http://www.xtranormal.com/watch/6995033/
- JabavuAdams 16y agoI find it interesting that with all this talk of scaling, and the specialized tools that go along with it, 4sq's service was essentially running on 2 servers. Maybe I only need 1 server to be profitable. Hmm...
- darkhorse 16y agoand out comes mongo's dirty little secret - you have to have enough ram in your boxes to hold not just all the data in ram, but all the indexes too, or it completely shits the bed. putting hundreds of gigs of ram in a box isn't cheap. are the foursquare folks considering rewriting with a traditional datastore like postgres and some memcached in front of it?
- hartror 16y agoErr you don't . . as the article states their usage and load pattern does. If you want the speed of keeping the DB in RAM you need the RAM available whether it is mongo, MySQL/NDB or memcache. (-1 for not RTFA)
- rit 16y agoThey migrated to MongoDB from Postgres. Specifically because they needed distributed sharding which Postgres didn't have. Jorge Ortiz, one of the developers at Foursquare said it best on twitter yesterday: "Baffled by the Mongo haters. If foursquare had stayed on Postgres, we would've had to write our own distributed sharding/balancing/indexing." "Two problems: 1) Our job is to build foursquare, not a database. 2) These things are hard. Odds are we would have had even more downtime." http://twitter.com/#!/jorgeortiz85/status/26563381834 http://twitter.com/#!/jorgeortiz85/status/26563381834 http://twitter.com/#!/jorgeortiz85/status/26563387808 http://twitter.com/#!/jorgeortiz85/status/26563387808 Which sums it up incredibly well. Your comment shows a scary lack of depth - this is the "dirty little secret" of ANY application that needs serious scalable data performance. You need to put your data in RAM where it is quickly accessible. As soon as you go to disk especially on a low I/O cloud box you are going to be in 'extremely slow' territory. This is why, even before the current NoSQL movement, everyone was using Memcached all over the place. Not because it was a fad but because they needed speed. SQL Databases have RAM caches as well which give you a pretty similar behavior. And big surprise - if you throw more memory at them - they run faster! Because they cache things in memory! In this case I believe that what Eliot was highlighting was that Foursquare's performance tolerance requirements were such that going to disk was not an optimal situation for them. There are plenty of applications out there using MongoDB without keeping it all in RAM; I've deployed and maintain several. The more memory you have certainly gives you better performance but it tends to be about a most frequently used cache. It never "Shit the bed" here to use your sophomoric vernacular. As the post states, it started having to go to the disk after it surpassed the memory threshold which slowed things down... nothing "crashed" as far as the description states. But a read/write queue backlog on any database is likely to exhibit the same behavior. Your postgres + memcached solution fixes what exactly? They would still need the same slabs of RAM to solve the problem, lest they go to disk on postgres and slow to a crawl the same way they did with MongoDB.
- sasidharm 16y agoI am not trying to start a debate of sql vs nosql, but i have an honest question. What if foursquare was using a standard RDBMS like Oracle? Would they have run into the same problem?
- andrewf 16y agoIn my experience with things like postgres, yes. If your working set (the data that you are touching regularly) grows past available RAM, you very rapidly switch from a system where disk access plays no part in most queries, to a system where it is the limiting factor. Because disks are so much slower, and having queries take only ten times as long can cause them to rapidly queue up, you end up with an unresponsive system. The appropriate response is almost always to turn off features to get the size of your working set back down, which buys you time to either alter the app to need less data, or get more RAM into the box.
- there 16y agoso, in short, a company relying entirely on cloud computing machines for storing its data, which is presumably being billed according to the memory usage of those machines, ran out of memory on them, and suffered a large amount of downtime as a result. mongodb had little to do with the problem, other than maybe it took longer than expected to migrate data to a third server. i'm baffled at how there could be no monitoring or reporting in place to catch that days or weeks ahead of time, let alone just 12 hours needed according to the mongodb developer, to fix the problem without downtime. it's such a fundamental thing to keep track of for a system designed entirely around storing a bunch of data in the memory of those 2(!) machines. i have more servers than foursquare and none of them do even a small fraction of the amount of processing that theirs do, and yet i have real-time bandwidth, memory, cpu, and other stats being collected, logged, and displayed, as well as nightly jobs that email me various pieces of information. nobody at foursquare ever even logged into those systems and periodically checked the memory usage manually? worse still, during all of this, the initial outage reports were blaming mongodb or saying the problem was unknown. even at that point nobody at foursquare realized that the servers were just out of memory? how did the developers come up with 66 gigabytes of ram to use for these instances in the first place? was there some kind of capacity planning to come up with that number or is it just a hard limit of EC2 and the foursquare developers maxed out the configuration?
- Devilboy 16y agoPlease take the time to capitalize your sentences, your otherwise great comment deserves it.
- jhugg 16y agoYou really should have considered a semi-colon over a comma. ;-)
- jemfinch 16y ago> how did the developers come up with 66 gigabytes of ram to use for these instances in the first place? The high-memory quadruple extra large (no, I'm not making that name up) instance offers 68.4GB of ram. Presumably they left the rest for the OS.
- look_lookatme 16y agoWhile I understand this is foremost a monitoring and architecture (mongo and 4sq side...) issue, but I'm curious how dangerous it is to run such IO dependent systems on EC2. Would the prolonged downtime (due to shard migration, etc) been as severe if they were running on hardware? What if these two servers were on SSD RAID 0?
- dirkgadsden 16y agoThe duration of the downtime seems to have been dependent on not just disk throughput, but also network and CPU throughput. Eliot did mention that a large portion of the downtime was partly caused by the slowness of EBS. Making a rough estimate, I think the downtime would probably be about 2/3 of how long it was if it had been on an SSD RAID. However, then you run into the issue of having to maintain your own servers; which, judging by the fact that they are using EC2 and EBS so heavily, Foursquare does not have any desire whatsoever to manage their own infrastructure.
- look_lookatme 16y agoIt's an interesting angle if that's the reason they are using EC2. Dedicated hosting might be more expensive, but hardware managed is pretty efficient at this point. Also, given the vast number of EC2 instances they could afford, it seems counter intuitive to be running only two mongrel shards. If you are that stingy about spreading data around, you might as well be using dedicated.
- dirkgadsden 16y agoIt doesn't seem that it was necessarily stinginess of neglect, but rather stinginess imposed by their situation. Like the article said, it took hours to create a new shard, downtime that Foursquare definitely did not want.
- look_lookatme 16y agoWhoops, I meant "mongo" shard!
- jmulho 16y agoCan we clear up one issue? Sharding on userid -- bad or fine? I say fine. If your algorithm is "if userid between A and M then server0, else (N-Z) server1", then you could get unbalanced. But if your algorithm is "if random 50/50 hash of userid = 0 then server0, else (random 50/50 hash = 1) server1", then you are going to stay almost exactly balanced (law of large numbers). You would never, ever, get anywhere near a 60/40 imbalance. So the choice to use userid, if implemented correctly, is completely fine.
- ww520 16y agoA typical way to deal with uneven distribution in sharding is to do Hash(key) Mod P, where Hash is a cryptographic hash and P is the number of partitions. This way the hash function would randomize the distribution of key.
- richardw 16y agoHow much can using SSD's for paging help with this? Or to just serve the reads, since they're excellent at random access - surely typical of Foursquare's usage?
- nailer 16y agoTo fix Google Group's unnecessary use of monospaced fonts so you can read this: http://lab.arc90.com/experiments/readability/ http://lab.arc90.com/experiments/readability/
- piramida 16y agoSo some people still use built-in memory allocation for large-scale memory-intense projects like mongodb eh? Pity that, they should have looked at why exactly memcached (and any other sane piece of memory-write-erase intensive application) does slab allocations.
- robryan 16y agoThis may be a little naive seeing as I haven't used MongoDB or know anything about foursquare architecture, but wouldn't a better result have been just to drop requests that were hitting the disk? Sure some people would have been getting a sub optimal experience but arguably it's a lot better than not having your service up at all.
- sofuture 16y agoFrom my understanding, that 'routing' decision would have to take place inside of Mongo -- null routing DDoS packets on a network is one thing, dropping certain database requests on the fly inside a database server sounds a littttle bit tricker.
- dedward 16y agoTunable concurrency control for the number of incoming requests would have been good - before they even hit mongo. Then you can just dial things down a bit (making customers wait a bit) and let the system recover, and tune things back to the point of optimal behaviour, rather than letting things overload. (You set your concurrency limits at a point where you know the system is just beginning to slow down but is still performing satisfactorily)
- kvs 16y agoLooks like there needs to be some data engineering to be done to back memory up with disk storage. I think you guys, by now, have enough information about users and checkins to create algorithms that can determine "hot" checkins vs. "cold" checkins as and when they come in. Because not every user is the same, a more user-centric view can definitely help you come up with a more scalable (or graciously failing) system. For example, I don't checkin as much as some of my friends who checkin 10-15 times a day. So, there is no point in giving my checkins as much priority as those busy folks.