5 ms·
so, in short, a company relying entirely on cloud computing machines for storing its data, which is presumably being billed according to the memory usage of tho
by there 16y ago
so, in short, a company relying entirely on cloud computing machines for storing its data, which is presumably being billed according to the memory usage of those machines, ran out of memory on them, and suffered a large amount of downtime as a result. mongodb had little to do with the problem, other than maybe it took longer than expected to migrate data to a third server.
i'm baffled at how there could be no monitoring or reporting in place to catch that days or weeks ahead of time, let alone just 12 hours needed according to the mongodb developer, to fix the problem without downtime. it's such a fundamental thing to keep track of for a system designed entirely around storing a bunch of data in the memory of those 2(!) machines. i have more servers than foursquare and none of them do even a small fraction of the amount of processing that theirs do, and yet i have real-time bandwidth, memory, cpu, and other stats being collected, logged, and displayed, as well as nightly jobs that email me various pieces of information. nobody at foursquare ever even logged into those systems and periodically checked the memory usage manually?
worse still, during all of this, the initial outage reports were blaming mongodb or saying the problem was unknown. even at that point nobody at foursquare realized that the servers were just out of memory?
how did the developers come up with 66 gigabytes of ram to use for these instances in the first place? was there some kind of capacity planning to come up with that number or is it just a hard limit of EC2 and the foursquare developers maxed out the configuration?
- Devilboy 16y agoPlease take the time to capitalize your sentences, your otherwise great comment deserves it.
- jhugg 16y agoYou really should have considered a semi-colon over a comma. ;-)
- jemfinch 16y ago> how did the developers come up with 66 gigabytes of ram to use for these instances in the first place? The high-memory quadruple extra large (no, I'm not making that name up) instance offers 68.4GB of ram. Presumably they left the rest for the OS.
- NIGGERFUCKER 16y agoHOLY FUCKING SHIT WHAT A CRAZY NAME
- hartror 16y agoExactly. How the hell does a fast growing well funded 24 employee startup NOT have load monitoring on their database servers! Pay the 10c the hour for a micro EC2 instance and run Zabbix or one of the half dozen other awesome monitoring packages out there.
- binspace 16y agoOther things get prioritized.
- hartror 16y agoLike not having a massive downtime that gives their users a reason to try out some massive social network's offering that has already started eating their lunch? Oh wait no they didn't prioritize that. My point is, and perhaps the down vote indicates I didn't make it well, that monitoring and alerts are extremely important for any startup. They admit as much in the article, had they known sooner there wouldn't have been any down time. It isn't an excuse to say it got lost in the crush (not that they are trying to use it as an excuse, they're being very upfront and admitting fault). We have neither the load or the team they do and we have comprehensive monitoring and alerts across our infrastructure as part of the SOE that each server gets. Perhaps we prioritized monitoring higher than them as we are a small team? And I bet they have prioritized monitoring now. Now where did I put the key to that barn door?
- metageek 16y agoNow where did I put the key to that barn door? It's in the saddlebags.
- binspace 16y agoHumans don't judge probability and risk very well. It's hard to forsee the future and problems. Hindsight is 20/20. My bag of cliches is exhausted. Maybe there were other things the engineers were working on that brought definite (as in 100%) benefit. Maybe there's a good product/consulting opportunity here. Everybody needs monitoring, but it takes time and resources. I agree, that they are focusing on it now, because it happened.
- datums 16y agoHopefully they get around to deploying open source monitoring with this hire http://foursquare.jobscore.com/jobs/foursquare/operations-engineer/dWqCdiYX8r34uheJe4bk1X http://foursquare.jobscore.com/jobs/foursquare/operations-en...
- pht 16y ago> how there could be no monitoring or reporting in place No. off Developers: x No. of Sysadmins: 0 It's a different job, often with a different career path, and a very differnt attitude. Though some of us get to be be both Code Monkey and the Scary Monk.