47 ms·
More details about the October 4 outage
- halotrope 5y agoIt is completely logical but still kind of amazing, that facebook plugged their globally distributed datacenters together with physical wire.
- packetslave 5y agoWhat do you imagine other companies use to connect their datacenters?
- meepmorp 5y agoUh... the cloud?
- zht 5y agoisn't that still over wires?
- I_AM_A_SMURF 5y agoOther people's wires :-)
- rblion 5y agoWas this a way to delete a lot of evidence before shit really hit the fan?' After reading this, I can't help but feel this was a calculated move. It gives FB a chance to hijack media attention from the whistleblower. It gives them a chance to show the average peson, 'hey, we make mistakes and we have a review process to improve our systems'. The timing is too perfect if you ask me.
- joemi 5y agoI'm not usually that cynical, but the timing of it combined with facebook's lengthy abusive relationship with customers' privacy (and what kind of company morals that implies) makes me think that it's definitely a possibility.
- rblion 5y agoThe testimony in front of congress and the reaction is just making me feel even more like this was a calculated move or internal sabotage.
- FridayoLeary 5y agoInviting further whistleblowing.
- cnst 5y agoNote that contrary to popular reports, DNS was NOT to blame for this outage — for once DNS worked exactly as per per the spec, design and configuration: > To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection.
- karmakaze 5y agoNot the first cause, but involved. Before reading I expected to see some combination of (1) automation (2) DNS (3) BGP. I didn't expect to see all three and the special automatically disconnect the backbone from the internet with no other way for senior tech staff to get to the backbone, not even a secure dial-up console. I think the general lesson here is for each thing you automate, assume that it can act in error and have another manual way to do what the automatic action prevents.
- rvnx 5y agoIt's a no apologies messages: "We failed, our processes failed, our recovery process only partially worked, we celebrate failure. Our investors were not happy, our users were not happy, some people probably ended in physically dangerous situations due to WhatsApp being unavailable but it's ok. We believe a tradeoff like this is worth it." - Your engineering team.
- rPlayer6554 5y agoLiterally the first line of their first article.[0] This article is just a technical explanation. [0] https://engineering.fb.com/2021/10/04/networking-traffic/outage/ https://engineering.fb.com/2021/10/04/networking-traffic/out...
- jffry 5y agoYesterday's blog post (discussion here: https://news.ycombinator.com/item?id=28754824 https://news.ycombinator.com/item?id=28754824) was a direct apology. Should that be repeated in a somewhat more technical discussion of why it happened?
- kzrdude 5y agoWe (the users) are the product anyway, they can only apologize to themselves for missing out on making money yesterday.
- mumblemumble 5y ago> Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command. I'm so glad to see that they framed this in terms of a bug in a tool designed to prevent human error, rather than simply blaming it on human error.
- cecilpl2 5y agoHuman error is a cop-out excuse anyway, since it's not something you can fix going forward. Humans err, and if a mistake was made once it could easily be made again.
- Cthulhu_ 5y agoI wouldn't be surprised if that tool was a shell script with a mistyped conditional somewhere, I really dislike shell scripting.
- infofarmer 5y agoAs opposed to what? Sixteen pages of boilerplate Java/Python?
- mumblemumble 5y agoI wouldn't conflate Java and Python in the boilerplate camp. Python can be very boilerplate-y, but it tends to only happen in the hands of Java developers. That said, even clean, idiomatic Python isn't as terse as sh. It also isn't as terse as perl. Many would argue that's a good thing. The optimum point for readability isn't found at either of the extremes. Not entirely unlike how the most readable way of writing English is neither shorthand nor blackletter.
- xphos 5y agoI don't know the -f in rm -rf isn't a bug xD. I feel sorry for the poor engineer who fat fingered the command. It definitely highlights an anti-pattern in the command line but the fact that, that singular console had the power to effect the entire network highlights an "interesting" design choice indeed.
- samhwr 5y agoIt _affected_ the whole network, and _effected_ the whole outage ;)
- ThinkBeat 5y agoI am curious about something. It has been quite a while since i had any job that required me to think about DCs. Back in the day we would have a setup of regular modems. If all hell broke loose, then we could dial up a modem and have access to the system. It was not fast, and it was a pain, but we could get access that way. (I am skipping a lot of steps. There was heavy security involved in getting access) I guess landlines might not be an option anymore??
- nabakin 5y agoYesterday's post from Facebook https://engineering.fb.com/2021/10/04/networking-traffic/outage/ https://engineering.fb.com/2021/10/04/networking-traffic/out...
- dmoy 5y agoInteresting bit on recovery w.r.t. the electrical grid > flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems ... I wish there was a bit more detail in here. What's the worst case there? Brownouts, exploding transformers? Or less catastrophic?
- kube-system 5y agoLikely tripping breakers or overload protection on UPSes? Often PDUs used in a rack can be configured to start servers up in a staggered pattern to avoid a surge in demand for these reasons. I'd imagine there's more complications when you're doing an entire DC vs just a single rack, though.
- Johnny555 5y agoI don't see how suddenly running more traffic is going to trip datacenter breakers -- I could see how flipping on power to an entire datacenter's worth of servers could cause a spike in electrical demand that the power infrastructure can't handle, but if suddently running CPU's at 100% trips breakers, then it seems like that power infrastructure is undersized? This isn't a case where servers were powered off, they were idle because they had no traffic. Do large providers like Facebook really provision less power than their servers would require at 100% utilization? Seems like they could just use fewer servers with power sized at 100% if their power system going to constrain utilization anyway?
- kube-system 5y agoI don't know the answer. But it's not too uncommon, in general, to provision for reasonable use cases plus a margin, rather than provision for worst case scenario.
- dfcowell 5y agoAll of the components in the supply chain will be rated for greater than max load, however power generation at grid scale is a delicate balancing act. I’m not an electrical engineer, so the details here may be fuzzy, however in broad strokes: Grid operators constantly monitor power consumption across the grid. If more power is being drawn than generated, line frequency drops across the whole grid. This leads to brownouts and can cause widespread damage to grid equipment and end-user devices. The main way to manage this is to bring more capacity online to bring the grid frequency back up. This is slow, since spinning up even “fast” generators like natural gas can take on the order of several minutes. Notably, this kind of scenario is the whole reason the Tesla battery in South Australia exists. It can respond to spikes in demand (and consume surplus supply!) much faster than generator capacity can respond. The other option is load shedding, where you just disconnect parts of your grid to reduce demand. Any large consumers (like data center operators) likely work closely with their electricity suppliers to be good citizens and ramp up and down their consumption in a controlled manner to give the supply side (the power generators) time to adjust their supply as the demand changes. Note that changes to power draw as machines handle different load will also result in changes to consumption in the cooling systems etc. making the total consumption profile substantially different coming from a cold start.
- HenryKissinger 5y ago> During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally Imagine being this person. Tomorrow on /r/tifu.
- savant_penguin 5y agoHahahahahahahaha
- kabdib 5y ago"It was the intern" is only cute once.
- Barrin92 5y agoof course if one person can knock down an entire global system through a trivial mistake the problem is obviously not the person to begin with, but the architecture of the system.
- flutas 5y agoOr the fact that there was a bug in the tool that should have prevented this. > Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command.
- bastardoperator 5y agoSo this is really two peoples fault. One for issuing the command the other for introducing the bug in the audit tool.
- colechristensen 5y agoNot really, there’s essentially always a button that blows everything up. Catastrophic failures usually end up being a large set of safety systems malfunctioning which would otherwise prevent the issue when that button is pressed. But yes, for these types of problem, the ultimate fault is never “that guy Larry is an idiot”, it takes a large team of cooperating mistakes.
- Hokusai 5y ago> One of the jobs performed by our smaller facilities is to respond to DNS queries. DNS is the address book of the internet, enabling the simple web names we type into browsers to be translated into specific server IP addresses. What is the target audience of this post? It is too technical for non-technical people, but also it is dumbed down to try to include people that does not know how the internet works. I feel like I'm missing something.
- thirtyseven 5y agoWith an outage this big, even a post for a technical audience will get read by non-technical people (including journalists), so I'm sure it helps to include details like this.
- shadofx 5y agoTeenagers who are responsible for managing the family router?
- Florin_Andrei 5y ago> Those data centers come in different forms. It's like the birds and the bees.
- eli 5y agoThe media. This was a huge international story.
- blowski 5y ago
- jbschirtzs 5y ago"The Devil's Backbone"
- secguyperson 5y ago> a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally From a security perspective, I’m blown away that a single person apparently had the technical permissions to do such a thing. I can’t think of any valid reason that a single person would have the ability to disconnect every single data center globally. The fact that such functionality exists seems like a massive foot-gun. At a minimum I would expect multiple layers of approval, or perhaps regionalized permissions, so that even if this person did run an incorrect command, the system turns around and says “ok we’ll shut down the US data centers but you’re not allowed to issue this command for the EU data centers, so those stay up”.
- chomp 5y agoSo someone ran "clear mpls lsp" instead of "show mpls lsp"?
- bryan_w 5y agoSeems like it. It's kinda like typing hostname and accidentally poking your yubikey (Not that I've done that...) or the date command that both let's you set the date and format the date
- fragmede 5y agoykpersonalize -u -1 -o -fast-trig (or some variation thereof)
- divbzero 5y agoFor context, parent comment is trying to decipher this heavily-PR-reviewed paragraph: > During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally. Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command.
- jauer 5y agos/assess/augment/g
- slim 5y agoMy guess : he executed the command from shell history. Commands look similar enough and he hit Enter too quickly
- dx034 5y agoBut why would the clear command be in their history?
- deleted 5y ago
- vitus 5y agoGoogle had a comparable outage several years ago. https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19... This event left a lot of scar tissue across all of Technical Infrastructure, and the next few months were not a fun time (e.g. a mandatory training where leadership read out emails from customers telling us how we let them down and lost their trust). I'd be curious to see what systemic changes happen at FB as a result, if any.
- silicon2401 5y ago> leadership read out emails from customers telling us how we let them down and lost their trust). That's amazing. I would never have expected my feedback to a company to actually be read, let alone taken seriously. Hopefully more companies do this than I thought.
- spaetzleesser 5y agoFrom my experience this is more done to make leadership feel better and deflect blame from their leadership.
- variant 5y agoI just read your comment out loud if it helps.
- hoppyhoppy2 5y agoFacebook also had a nearly 24-hour outage in 2019. https://www.nytimes.com/2019/03/14/technology/facebook-whatsapp-outage.html https://www.nytimes.com/2019/03/14/technology/facebook-whats... (or http://archive.today/O7ycB http://archive.today/O7ycB )
- vitus 5y agoTo expand on why this made me think of the Google outage: It was a global backbone isolation, caused by configuration changes (as they all are...). It was detected fairly early on, but recovery was difficult because internal tools / debugging workflows were also impacted, and even after the problem was identified, it still took time to back out the change. "But wait, a global backbone isolation? Google wasn't totally down," you might say. That's because Google has two (primary) backbones (B2 and B4), and only B4 was isolated, so traffic spilled over onto B2 (which has much less capacity), causing heavy congestion.
- deleted 5y ago[deleted]
- tigerlily 5y agoWe want ramenporn
- cesarb 5y agoContext for those who didn't see it yesterday: https://news.ycombinator.com/item?id=28749244 https://news.ycombinator.com/item?id=28749244
- henrypan1 5y agoWow
- TedShiller 5y agoDuring the outage, FB briefly made the world a better place
- i_like_apis 5y agothe total loss of DNS broke many of the internal tools we’d normally use to investigate and resolve outages like this. this took time, because these facilities are designed with high levels of physical and system security in mind. They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them. Sounds like it was the perfect storm.
- codebolt 5y agoApparently they had to bring in the angle grinder to get access to the server room. https://twitter.com/cullend/status/1445156376934862848?t=P5ua0Fk7iPT5g-05bgaA8w&s=19 https://twitter.com/cullend/status/1445156376934862848?t=P5u...
- jd3 5y agoWas this ever confirmed? NYT tech reporter Mike Isaac issued a correction to his previous reporting about it. > the team dispatched to the Facebook site had issues getting in because of physical security but did not need to use a saw/ grinder. https://twitter.com/MikeIsaac/status/1445196576956162050 https://twitter.com/MikeIsaac/status/1445196576956162050
- packetslave 5y agothat didn't happen (NY Times corrected their story)
- deleted 5y ago[deleted]
- mzs 5y agoor not https://twitter.com/MikeIsaac/status/1445196576956162050 https://twitter.com/MikeIsaac/status/1445196576956162050 https://twitter.com/cullend/status/1445212476652535815 https://twitter.com/cullend/status/1445212476652535815
- Ansil849 5y ago> The backbone is the network Facebook has built to connect all our computing facilities together, which consists of tens of thousands of miles of fiber-optic cables crossing the globe and linking all our data centers. This makes it sound like Facebook has physically laid "tens of thousands of miles of fiber-optic cables crossing the globe and linking all our data centers". Is this in fact true?
- colechristensen 5y agoLikely a mixture of bought, leased, and self laid fiber. This is not at all uncommon and basically necessary if you have your own data center.
- Closi 5y agoYes, although undoubtedly a lot of this is shared investment - https://www.businessinsider.com/google-facebook-giant-undersea-cables-internet-tech-2021-9 https://www.businessinsider.com/google-facebook-giant-unders... https://datacenterfrontier.com/facebook-will-begin-selling-wholesale-fiber-capacity/ https://datacenterfrontier.com/facebook-will-begin-selling-w...
- louwrentius 5y ago> Our primary and out-of-band network access was down Don't create circular dependencies.
- vitus 5y agoWith something as fundamental as the network, no way around it. - Okay, we'll set up a separate maintenance network in case we can't get to the regular network. - Wait, but we need a maintenance network for the maintenance network...
- packetslave 5y ago"Okay, we'll pull in a DSL line from a completely separate ISP for the out-of-band access." (guess what else is in that manhole/conduit?) "Okay, we'll use LTE for out-of-band!" (oops, the backhaul for the cell tower goes under the same bridge as the real network) True diversity is HARD (not unsolvable, just hard. especially at scale)!
- detaro 5y agoAlthough the difference here is that loosing connection and out-of-band for a single data center shouldn't be as catastrophic for Facebook, so your examples would be tolerable?
- packetslave 5y agoThat's the trick, though: if you don't do that level of planning for all of your datacenters and POPs (and fiber huts out in the middle of nowhere), it's inevitable that the one you most need to access during an outage will be the one where your OOB got backhoe'd. Murphy is a jerk.
- chasd00 5y agoheh i toured a large data center here in dallas and listened to them brag about all the redundant connectivity they had while standing next to the conduit where they all entered the building. One person, a pair of wire cutters, and 5 seconds and that whole datacenter is dark.
- nonbirithm 5y agoDNS seems to be a massive point of failure everywhere, even taking out the tools to help deal with outages themselves. The same thing happened to Azure multiple times in the past, causing complete service outages. Surely there must be some way to better mitigate DNS misconfiguration by now, given the exceptional importance of DNS?
- cnst 5y agoBut DNS didn't actually fail. Their design says DNS must go offline if the rest of the network is offline. That's exactly what DNS did. Sounds like their design was wrong, but you can't just blame DNS. DNS worked 100% here as per the task that it was given. > To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection.
- jaywalk 5y agoI'm not sure the design was even wrong, since the DNS servers being down didn't meaningfully contribute to the outage. The entire Facebook backbone was gone, so even if the DNS servers continued giving out cached responses clients wouldn't be able to connect anyway.
- Xylakant 5y agoDNS being down instead of returning an unreachable destination did increase load for other DNS resolvers though since empty results cannot be cached and clients continued to retry. This made the outage affect others.
- cnst 5y agoSource? DNS errors are actually still cached; it's something that has been debunked by DJB like a couple of decades ago, give or take: http://cr.yp.to/djbdns/third-party.html http://cr.yp.to/djbdns/third-party.html > RFC 2182 claims that DNS failures are not cached; that claim is false. Here are some more recent details and the fuller explanation: https://serverfault.com/a/824873 https://serverfault.com/a/824873 Note that FB.com currently expires its records in 300 seconds, which is 5 minutes. PowerDNS (used by ordns.he.net) caches servfail for 60s by default — packetcache-servfail-ttl — which isn't very far from the 5min that you get when things aren't failing. Personally, I do agree with DJB — I think it's a better user experience to get a DNS resolution error right away, than having to wait many minutes for the TCP timeout to occur when the host is down anyways.
- tantalor 5y agoSo it wasn't a config change, it was a command-of-death.
- bilater 5y agoI want to know what happened to the poor engineer who issued the command?
- pjscott 5y agoThe blog post is putting the blame on a bug in the tooling which should have made the command impossible to issue, which is exactly where the blame ought to go.
- bilater 5y agoStill I'd hate to be the first 'Why' of a multi-billion dollar outage :D
- chovybizzass 5y agoor the manager who told him to "do it anyway" when he raised concern.
- antocv 5y agoHe will be promoted to street dweller while his managers will fail up.
- runawaybottle 5y agoActually he was promoted to C-Level, CII, Chief Imperial Intern ‘For Life’. It’s a great accomplishment to be be fair, comes with a lifetime weekly stipend and access to whatever Frontend books/courses you need to be a great web developer. Will never touch ops again.
- jimmyvalmer 5y agotldr; a maintenance query was issued that inexplicably severed FB's data centers from the internet, which unnecessarily caused their DNS servers to mark themselves defunct, which made it all but impossible for their guys to repair the problem from HQ, which compelled them to physically dispatch field units whose progress was stymied by recent increased physical security measures.
- antocv 5y ago> caused their DNS servers to mark themselves defunct This is awkward for me too, why should a DNS server withdraw BGP routes? Design fail.
- chronid 5y agoNote those are anycast addresses, my guess is the DNS server gives out addresses for FB names pointing your traffic to the POP the DNS server is part of. If the POP is not able to connect to the rest of Facebook's network, the POP stops announcing itself as available and that DNS and part of the network goes away so your traffic can go somewhere else.
- yuliyp 5y agoIt's a trade-off. Imagine you have some DNS servers at a POP. They're connected to a peering router there which is connected to a bunch of ISPs. The POP is connected via a couple independent fiber links to the rest of your network. What happens if both of those links fail? Ideally the rest of your service can detect that this POP is disconnected, and adjust DNS configuration to point users toward POPs which are not disconnected. But you still have that DNS server which can't see that config change (since it's disconnected from the rest of your network) but still reachable from a bunch of local ISPs. That DNS server will continue to direct traffic to the POP which can't handle it. What if that DNS server were to mark itself unavailable? In that case, DNS traffic from ISPs near that POP would instead find another DNS server from a different POP, and get a response which pointed toward some working POP instead. How would the DNS server mark itself unavailable? One way is to see if it stopped being able to communicate with the source of truth. Yesterday all of the DNS servers stopped being able to communicate with the source of truth, so marked themselves offline. This code assumes a network partition, so can't really rely on consensus to decide what to do.
- Ansil849 5y ago> We’ve done extensive work hardening our systems to prevent unauthorized access, and it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity, but an error of our own making. I believe a tradeoff like this is worth it — greatly increased day-to-day security vs. a slower recovery from a hopefully rare event like this. From here on out, our job is to strengthen our testing, drills, and overall resilience to make sure events like this happen as rarely as possible. I found this to be an extremely deceptive conclusion. This makes it sound like the issue was that Facebook's physical security is just too gosh darn good. But the issue was not Facebook's data center physical security protocols. The issue was glossed over in the middle of the blogpost: > Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command. The issue was faulty audit code. It is disingenuous to then attempt to spin this like the downtime was due to Facebook's amazing physec protocols.
- temp_praneshp 5y agoNo one except you is trying to spin anything.
- Ansil849 5y agoGoing on at lengths about how the trade off between prolonged downtime and strict security protocols is a worthy trade off is erecting a nonsensical strawman, the literal definition of spinning a story. The key issue had nothing to do with Facebook's data center security protocols.
- temp_praneshp 5y agoNo one except you is erecting a nonsensical strawman.
- Ansil849 5y agoExcept, I quoted exactly where they were.
- Jamie9912 5y agoWhat would they have done if the whole data center was destroyed?
- detaro 5y agocontinue working form all other data centers, possibly without users really noticing.
- jasonjei 5y agoWill somebody lose a job over this?
- mman0114 5y agoThey shouldn't. If FB has a proper PMA culture in place, you figure out why the processes in place didn't work that allowed this kind of change to happen, so more testing, etc. Should be a blameless exercise.
- _moof 5y agoNot if FB has a halfway decent engineering culture. People make mistakes. They're practically fundamental to being a person. You can minimize mistakes, but any system that requires perfect human performance will fail.
- pm2222 5y agoI'm confused by this. Are the DNS servers inside the backbone, or outside? To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP advertisements. The end result was that our DNS servers became unreachable even though they were still operational. This made it impossible for the rest of the internet to find our servers.
- net4all 5y agoIf I understand it correctly they have DNS servers spread out at different locations. These locations are also BGP peering locations. If the DNS server at a location cannot reach the other datacenters via the backbone it stops advertising their IP prefixes. The hope is that traffic will then instead get routed to some other facebook location that is still operational.
- deleted 5y ago[deleted]
- cranekam 5y agoThe wording is a bit unclear (rushed, no doubt) but I expect this means the DNS servers stopped announcing themselves as possible targets for the anycasted IPs Facebook uses for its authoritative DNS [1], since they learned that the network was deemed unhealthy. If they all do that nobody will answer traffic sent to the authoritative DNS IPs and nothing works. [1] See "our authoritative name servers that occupy well known IP addresses themselves" mentioned earlier
- pm2222 5y agoThe anycasted IPs for DNS servers make sense to me and the BGP withdrawal too when the common cases are perhaps one or a few PoPs lost connectivity to the backbone/DC rarely every one of them fails at the time. I was hoping perhaps the DNS servers at PoPs can be improved by responding with public IPs for other PoPs/DCs and only when that is not available start the BGP withdrawal. Or I can presume the available DNSs at PoPs decrease over time and the remaining ones getting more and more requests until finally every one of them is cut off from the internet.
- deleted 5y ago[deleted]
- harshreality 5y ago> To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. No, it's (clearly) not a guaranteed indication of that. Logic fail. Infrastructure tools at that scale need to handle all possible causes of test failures. "Is the internet down or only the few sites I'm testing?" is a classic network monitoring script issue.
- jaywalk 5y agoI think you're misunderstanding. The DNS servers (at Facebook peering points) had zero access to Facebook datacenters because the backbone was down. That is as unhealthy as the network connection can get, so they (correctly) stopped advertising the routes to the outside world. By that point, the Facebook backbone was already gone. The DNS servers stopping BGP advertisements to the outside world did not cause that.
- harshreality 5y agoYou're talking about backend network connections to facebook's datacenters as if that's the only thing that matters. I'm talking about overall network connection including the internet-facing part. Facebook's infrastructure at their peering points loses all contact with their respective facebook datacenter(s). Their response is to automatically withdraw routes to themselves. I suppose they assumed that all datacenters would never go down at the same time, so that client dns redundancy would lead to clients using other dns servers that could still contact facebook datacenters. It's unclear how those routes could be restored without on-site intervention. If they automatically detect when the datacenters are reachable again, that too requires on-site intervention since after withdrawing routes FB's ops tools can't do anything to the relevant peering points or datacenters. But even without the catastrophic case of all datacenter connections going down, you don't need to be a facebook ops engineer to realize that there are problems that need to be carefully thought through when ops tools depends on the same (public) network routes and DNS entries that the DNS servers are capable of autonomously withdrawing.
- ghostoftiber 5y agoIncidentally the facebook app itself really handled this gracefully. When the app can't connect to facebook, it displays "updates" from a pool of cached content. It looks and feels like facebook is there, but we know it's not. I didn't notice this until the outage and I thought it was neat.
- divbzero 5y agoI’m curious if the app handled posts or likes gracefully too. Did it accept and cache the updates until it could reconnect to Facebook servers?
- ghostoftiber 5y agoI'm not a huge facebook user and I mostly use it to keep in touch with my parents. I did notice that trying to message them resulted in "message not delivered" (which gave me a prompt to retry) but I didn't try to post anything during the outage.
- deleted 5y ago[deleted]
- colanderman 5y agoWhatsApp failed similarly, but I thought it was a poor design decision to do so. Anyone waiting on communication through WhatsApp had no indication (outside the media) that it was unavailable, and that they should find a nother communication channel. Don't paper connectivity failures. It disempowers users
- jrockway 5y agoI think these features are designed for the timescales like "between subway stations which have cell service" or even "driving into a tunnel". In those cases, it seems totally appropriate to me.
- avh02 5y agowhatsapp used to have a "status" page in their app, they removed it years ago though. That said - yeah, i figured there's an issue when hard refreshing whatsapp web (sometimes has a kink that needs to be refreshed away) didn't work.
- cube00 5y ago> We’ve done extensive work hardening our systems to prevent unauthorized access, and it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity, but an error of our own making. I believe a tradeoff like this is worth it — greatly increased day-to-day security vs. a slower recovery from a hopefully rare event like this. If you correctly design your security with appropriate fall backs you don't need to make this trade off. If that story of the Facebook campus having no physical key holes on doors is true it just speaks to an arrogance of assuming things can never fail so we don't even need to bother planning for it.
- joshuamorton 5y agoCan you elaborate on this? There are always going to be security/reliability tradeoffs. Things that fail closed for security reasons will cause slower incident responses. That's unavoidable. Innovation can improve the frontier, but there will always be tradeoffs.
- cube00 5y agoSlower sure, but not five hour slow.
- randomswede 5y agoThe moment you need to start moving people around, you are into "hours" territory of recovery. You don't want the data centre staff to be able to change configurations (security), so once something requires hands-on changing, you are definitely into the "move people around" stage of recovery and it WILL be slow.
- fakeythrow8way 5y agoFor a lot of people in countries outside the US, Facebook _is_ the internet. Facebook has cut deals with various ISPs outside the US to allow people to use their services without it costing any data. Facebook going down is a mild annoyance for us but a huge detriment to, say, Latin America.
- byron22 5y agoWhat kind of bgp command would do that?
- 1vuio0pswjnm7 5y ago"Those translation queries are answered by our authoritative name servers that occupy well known IP addresses themselves, which in turn are advertised to the rest of the internet via another protocol called the border gateway protocol (BGP)." "To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection." Correct me if I am wrong, but here "DNS servers" means the computers, not the software running on them, i.e., each computer is running both DNS software and a BGP daemon. I am not aware of DNS server software that disables BGP advertisements but a BGP daemon could do it. For example, a BGP daemon like ExaBGP can execute a DNS query, check the output and disable advertisements if the query fails. https://github.com/Exa-Networks/exabgp https://github.com/Exa-Networks/exabgp
- nbm 5y agoPerhaps the right word is “system” - the combination of software (DNS protocol server software, health checker, BGP agent, and so forth) involved in making the DNS service available (or not, in this case). These could be running on the same computer, or separate ones if you are particularly imaginative. Unfortunately, “DNS system” means something very different than if you said “load balancing system”, so “server” is simpler. (Usual disclaimer: work at FB, even on this exact stuff, but not representing it.)
- Animats 5y agoYou can see the security/reliability tradeoff problem here. You need a control plane. But what does it run over? Your regular data links? A problem if it also controls their configuration. Something outside your own infrastructure, like a modest connection to the local ISP as a backup? That's an attack vector. One popular solution is to keep both the current and previous generation of the control plane up. Both Google and AT&T seem to have done that. AT&T kept Signalling System 5 up for years after SS7 was doing all the work. Having two totally different technologies with somewhat different paths is helpful.
- eddieroger 5y agoThe post mentioned that the out-of-band network was also down, and I’m curious what that entails and how it was also impacted. They must not have been on external DNS or had static IPs to access recovery. I’m sure they won’t share more than this now, but I’d sure love to hear more about the OOB access.
- kaustubhvp 5y agoWhy were all BGP routes advertised from same set of servers in same DC which were pointed by ALL fb owned domains?
- randomswede 5y agoI don't think they were. But, without knowing more specifically what happened, it is hard to speculate. Some of the speculation I have seen was that this was an attempt to cut connections to one internet exchange (IXP) point, possibly resulting in cutting connection to all IXPs. But ,as I said, this is speculation. It'd take a more thorough understanding of how the automation looked prior to the incident, the change hat was trying to be made, and eth like, to say something sensible.
- perryizgr8 5y ago> Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems to caches at risk. Interesting that they think fluctuations of tens of megawatts would risk electrical systems. If the equipment was handling that much continuous load, wouldn't it also easily handle the resumption of the same load? Also I totally did not understand how power usage would affect caches.
- lovecg 5y agoContinuous load is very different from a sudden 0 to 100 increase.
- perryizgr8 5y agoWhen facebook is directly peering to so many other ASes, why would they not have static routes in place for those direct links? Why run BGP for that? It's not like there is going to be a better route than the direct link. If the link goes down, then you can rely on BGP to reroute.
- Vaderv 5y agoIt was a hacker group