6 ms·
Details on today's Facebook outage
- ergo98 16y agoThey built their own custom DNS server? Most of the failures that people encountered were a failure to contact the nameserver itself. Perhaps in the rush to try to fix it someone screwed up that as well.
- brown9-2 16y agoPerhaps the DNS issue was their way of shutting off the site.
- mbreese 16y agoIt seems like that was their attempt to "shutdown" the site. It's a pretty effective way too... The easiest way to stop the stampede was to drop off the net completely. By tweaking their dns, they were able to give themelves enough room to breathe. Then they could slowly start to bring people back. That's just a guess. But from their post, it seems reasonable.
- dangrossman 16y agoOr maybe disabling DNS was how they purposely took down the site, then slowly let people back in, as they said.
- ergo98 16y agoDNS was never mentioned in that post. Further it's a curious way of controlling flow, though I suppose in a moment of panic...if nothing else works...
- schrep 16y agoThis was a result of us throttling traffic to the site - not the original outage.
- cageface 16y agoThis certainly won't be the first time that a system designed to increase uptime actually reduces it. I've seen a lot of "redundant" systems that are actually less reliable than simple standalones thanks to all the extra complexity of clustering. I guess at Facebook's scale you have to build in fallbacks but this is a reminder that you can easily do more harm than good.
- ora600 16y agoSometimes cluster systems are mistaken for high-availability solutions while they are actually load-balancing solutions and can decrease availability due to added complexity and dependencies.
- olefoo 16y agoSometimes designers and builders of a system make decisions about tradeoffs between availability and scaling and error recovery that they don't understand until after the system is in the field. Any sufficiently complex system becomes vulnerable to unexpected interactions between subsystems especially when one or more subsystems fails catastrophically. http://en.wikipedia.org/wiki/System_accident http://en.wikipedia.org/wiki/System_accident has more on the topic, but you do want to read Perrow's book http://books.google.com/books?id=VC5hYoMw4N0C&printsec=frontcover&dq=Charles+Perrow#v=onepage&q&f=false http://books.google.com/books?id=VC5hYoMw4N0C&printsec=f... Normal Accidents.
- smakz 16y agoVery true - the complexity of the system also comes into the equation when things go wrong and it takes real people to figure out why an outage is happening. Highly complex systems imply longer debugging time, and at a certain point a theoretically lower up time can give you higher practical up time just because engineers can actually understand and debug it.
- jasonwatkinspdx 16y agoThe other thing is that systems that cut across your architecture need to be handled with utter paranoia. They'll be the coupling points that cause global outages.
- itistoday 16y agoThat was an excellent description of the problem. Too excellent, it seems, for many of the commenters. :-p
- AgentConundrum 16y agoFor a second there, while reading the comments, I wasn't entirely sure I hadn't accidentally been redirected to a redesigned YouTube. Interesting article, terrible comments.
- duck 16y agoYeah, I don't know how regular users find posts like this... but that was way over everyones head.
- pquerna 16y agoAgree, ideally they should of made two posts, one like this, and another for the 'normal' human being.
- Malakin 16y agoThis is how http://twitter.com/facebook http://twitter.com/facebook (how ironic)
- docgnome 16y agoIronic? I don't really think so. They aren't exactly competitors.
- moe 16y agoAt least one actually understood it quite well and translated the issue adequately, and hilariously, for the remaining audience: Angee Lening Marvin: the server was like a dog chasing its tail... it kept going in circles, but never caught it. They basically had to hold the tail for the dog so he could bite a flea on it. :) LOL
- sethwartak 16y agoMy favorite comments on fb page: Stick with Mysql * * * * If it aint broke, dont fix it! Me too Melissa, and it's out there in the media that a group of hackers caused the problem, is this true, Mr Robert Johnson? PLEASE !!!! WHAT CAN YOU DO TO HELP THIS FROM EVER HAPPENING AGAIN???????????????? PLEASE!!!!!!!!!!!!!!! CANDY Kip da updates comin' Did anyone get a message like I did about someone trying to access your account from another state?
- riffer 16y agoYeah, I noticed that too, it's kind of sad when you think about it
- catshirt 16y ago"This means that Facebook is a database-driven app :D LOL , just refactor FB, you know it is a mess."
- deleted 16y ago[deleted]
- swaarm 16y agoI actually came across a pretty good comment which explains it (to the layman) pretty well: "Marvin, the server was like a dog chasing its tail...it kept going in circles, but never caught it. They basically had to hold the tail for the dog so he could bite a flea on it. :) LOL"
- deleted 16y ago[deleted]
- davidu 16y agoThis is known generally as the "Thundering Herd" problem: The thundering herd problem occurs when a large number of processes waiting for an event are awoken when that event occurs, but only one process is able to proceed at a time. After the processes wake up, they all demand the resource and a decision must be made as to which process can continue. After the decision is made the remaining processes are put back to sleep, only to wake up again to request access to the resource. This occurs repeatedly, until there are no more processes to be woken up. Because all the processes use system resources upon waking, it is more efficient if only one process is woken up at a time. This may render the computer unusable, but it can also be used as a technique if there is no other way to decide which process should continue (for example when programming with semaphores). Though the phrase is mostly used in computer science, it could be an abstraction of the observation seen when cattle are released from a shed or when wildebeest are crossing the Mara River. In both instances, the movement is suboptimal. From: http://en.wikipedia.org/wiki/Thundering_herd_problem http://en.wikipedia.org/wiki/Thundering_herd_problem
- ora600 16y agoI'm often encountering systems that are designed for very short queries processing small numbers of rows where the number of connections from the app servers is configured to be far greater than the number of CPUs. Since those systems are doing very little IO, configuring connection pools to start with much more connections than the DB has CPUs and to add more if the connections are busy (i.e. DB is getting slower, probably because it is highly loaded), is guaranteed to cause resource contention on the DB and escalate the issue in case something goes wrong. Its a total waste and yet the most common configuration in the world.
- Fluxx 16y agoThis is a common pitfall when people start using Memcache. Memcache puts less load on the DB, which means you can keep the same DB hardware and scale up your pageviews...until Memcache craps out and all that load is now back on the DB again :(
- VladRussian 16y ago
- praeclarum 16y agoSo umm no mention of the 4chan DOS attack? I mean, not that I hang out there or anything, but a friend told me that they organized an attack. You'know. Jus sayin. /b/ye
- ramidarigaz 16y agoDOSing Facebook? It would require a huge effort to create a measurable increase in Facebook's traffic.
- staunch 16y agoNot if they found some expensive requests to use and/or if they have access to large botnets.
- whatusername 16y agosure - if you found some really really expensive requests perhaps. As has been stated -- It's pretty hard to put a bump in FB's traffic. The more likely explanation is that someone on 4chan noticed FB was down -- and then started talking about DOSing it.
- praeclarum 16y agoCompletely possible. Except the posts to call for the attack were made prior to the world-known outages. But I can't argue your point. I just found it curious that no one mentioned the planned attack.
- 146 16y agoThe Russians were barely able to slow down Facebook back in 2009. Unless someone on 4chan has a bigger botnet and just decided for lulz to DDoS Facebook, 4chan would not even come close to generating enough traffic to register on the #2 website on the Internet.
- alphabeat 16y agoAnything is possible thought right? If you target the right weakness?
- jrockway 16y agoThe text of the article is: "You are using an incompatible web browser. Sorry, we're not cool enough to support your browser. Please keep it real with one of the following browsers:" This is why I don't use Facebook. It's not 1990 anymore. You don't need User-Agent sniffing.
- kahawe 16y agoMy own theory what actually happened: http://www.reddit.com/r/fffffffuuuuuuuuuuuu/comments/di0v0/so_facebook_is_down_huh/ http://www.reddit.com/r/fffffffuuuuuuuuuuuu/comments/di0v0/s...