19 ms·
Understanding how Facebook disappeared from the internet
- andp97 5y agoNo one. Cloudflare that write a report of some else network outage! AWESOW COMPANY!!!
- phoe-krk 5y agoDo you really think that Facebook is going to write one right now? It's good that we have some coverage from companies that have some stake in the game because they are also affected by the outage, even if only partially.
- Wolfspirit 5y agoEven if they wanted to write one they have no way to host it :D
- powera 5y agoFacebook has to make some public statement; the shareholders will demand it. I expect the detail level to be roughly "an automated system pushed a broken configuration"; that is to say, there probably won't be any interesting information at all for the Hacker News crowd. I doubt that this was caused by "hackers" or "hostile governments" or "dissident employees upset about Facebook privacy issues", and also doubt that Facebook would admit such if it were true unless they were legally required to do so.
- 0des 5y ago>Facebook has to make some public statement History has shown us they can give us zero response, or an incorrect response, and we (via our representatives) will accept it and continue living life as before.
- bilbo0s 5y agoThe fact is, it's altogether likely that they could be legally require NOT to make such a statement outlining the cause if it was a hostile actor. I've felt a distinct change recently. The US government is not messing around about cyber security anymore. The guys with the blue windbreakers show up, I'd pretty much say "yes, sir." Of course, I don't have FB's power, but I don't think it matters.
- atok1 5y agoI have some fresh boot soup I can offer you? With a side of steel-toe.
- shyn3 5y agoFacebook has a special ownership structure which means the shareholders have no input as their shares do not have voting power.
- cyanydeez 5y agonow adays, sharehooders maybe be just as quiet as everyone at facebook knowing making a big deal could lose them value, and theyd rather sell. shareholders aint much these days of cancerous growth
- clemenspw 5y agoThanks, great write up!
- deleted 5y ago[deleted]
- throwdecro 5y agoThis is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?
- kube-system 5y agoGiven my experience with DNS issues, I am guessing that they are running into dependencies along the way that assume/require DNS be available to function.
- theshadowknows 5y agoYeah as part of my job I often have to work with our DNS team to provision say a subdomain or get some domain verified. They’ve got like…three people…trying to service thousands of teams across the enterprise. I do not envy their job at all.
- EricE 5y agoYour guys need BP Diamond IP http://www.diamondipam.com http://www.diamondipam.com It's an IP address management (IPAM) solution that also just happens to be a fantastic, federated (if you want) DNS management system too. Indeed a previous org I worked at bought it strictly to tame the DNS beast - local sys admins could control DNS for their subnets but not affect anything else. If we wanted, we could have had approval processes on top of the change requests - the system supported that too. I think the security teams finally woke up to the IP address management functionality and were slowly starting to integrate that into the rest of the infrastructure - but I was leaving around then. It was a fantastic system. One of the best hierarchical role-based access control systems in an application I have ever seen; the granularity was amazing yet it was easy to understand/administer. Not an easy trick!
- bink 5y agoWith routing it's even worse than that. If they had no out-of-band method to connect to these routers and they botched the routing config then they had no way to route any traffic to them at all. At least with DNS you can still connect to the IPs. I would find it a bit surprising if Facebook didn't have OOB access to their data centers, however.
- nabaraz 5y agoIs there a way to secure BGP? Some kind of BGP table backup and restore?
- Wolfspirit 5y agoI think that it is possible to restore the previous state but the question is, if it makes sense. When do you decide that it was a failure? Facebook explicitly (even so automatically) told all others that they shouldn't use that routes anymore. When it comes to Facebooks side I guess they do have backups of their BGP config. Applying them (probably remotely) however seems to be harder then expected when the whole infrastructure is down.
- cyanydeez 5y agoreading krebs suggest they couldnt do anything because all the security ran through the same computers keeping facebook.com they basically ran into a footgun
- mnordhoff 5y ago"Because of this Cloudflare’s 1.1.1.1 DNS resolver could no longer respond to queries asking for the IP address of facebook.com or instagram.com." The instagram.com zone itself uses a third-party DNS service and didn't go down. (But e.g. www.instagram.com is a CNAME to a zone on FB DNS.)
- yftsui 5y agoThat’s pretty much why during the downtime visit instagram.com showed a 503 from AWS instead.
- mschuster91 5y agoWonder why they're still using AWS given that FB operates its own data centers...
- mnordhoff 5y agoNo idea. I'd speculate that it's some kind of historical reasons from before FB acquired IG.
- hinkley 5y agoCould be that nobody thought to ask. If it was an oversight that'll be rectified shortly, now that everyone in the company knows about it.
- hansel_der 5y agoprobably a testimony to the cost of aws lock-in
- jgrahamc 5y agoThanks. I'll correct that.
- AlbertCory 5y agoIt sorta sounds like Facebook is Too Big To Fail. Yet another reason to dismantle it.
- literallyWTF 5y agoI would rather purge it from this planet and throw Zuck into jail.
- deleted 5y ago[deleted]
- literallyaduck 5y agoIs it harmful? https://www.theguardian.com/technology/2021/sep/29/facebook-hearing-latest-children-impact https://www.theguardian.com/technology/2021/sep/29/facebook-... Did Facebook lie to congress? https://www.forbes.com/sites/zakdoffman/2019/03/02/facebook-caught-lying-about-spying-on-teenage-users-data-yet-again/?sh=3f69f3d64288 https://www.forbes.com/sites/zakdoffman/2019/03/02/facebook-... Also short comments without discussion and evidence do poorly. I believe it should be halted for public safety which is well within the rights and capacity of our government. I believe a fair trial should happen before anyone receives punishment, but I don't believe we could find an impartial jury as Facebook is ubiquitous.
- literallyWTF 5y agoWhy would I care if my short comment does poorly or not? Everything’s always a 10 page “discussion” on this site fuck me
- literallyWTF 5y agoOnce again, denizens of hacker news, blow me
- deleted 5y ago[deleted]
- threeseed 5y ago
- robaato 5y agoFrom article: [This chart shows] the availability of the DNS name 'facebook.com' on Cloudflare's DNS resolver 1.1.1.1. It stopped being available at around 15:50 UTC and returned at 21:20 UTC.
- Wolfspirit 5y agoSeems like they're back online
- ruoso 5y ago> ... but as of 22:28 UTC Facebook appears to be ... Someone assumed London==UTC, when London is 1 hour ahead :) that was actually 21:28 UTC
- interestica 5y agoNo matter what time of year it is, people tend to use 'EST' for 'Eastern Time' even when we might be in Eastern Daylight Time rather than Standard. It's especially annoying when dealing with multiple countries that may or may not be using Daylight Saving Time.
- Wolfspirit 5y agoEven google isn't quite sure about the summer time. Not sure if that is just a Google German thing... A few weeks ago I tried to find out what the current time in CET is. Asking google for "CET" gave me: "23:27 CET". Asking google for "CET time" (I know that "time" is twice in this case) gave me "00:27 CET". The last one is wrong and should be CEST or even more correct would be just the same result for CET as I asked for
- imglorp 5y agoI am always forgetting the polarity. Seeing the cut-overs listed helps me: $ zdump -v Europe/Berlin | grep 2021 Europe/Berlin Sun Mar 28 00:59:59 2021 UT = Sun Mar 28 01:59:59 2021 CET isdst=0 gmtoff=3600 Europe/Berlin Sun Mar 28 01:00:00 2021 UT = Sun Mar 28 03:00:00 2021 CEST isdst=1 gmtoff=7200 Europe/Berlin Sun Oct 31 00:59:59 2021 UT = Sun Oct 31 02:59:59 2021 CEST isdst=1 gmtoff=7200 Europe/Berlin Sun Oct 31 01:00:00 2021 UT = Sun Oct 31 02:00:00 2021 CET isdst=0 gmtoff=3600
- Jamie9912 5y agoIn this case, Google is providing a wrong answer to not confuse people. People don't understand that a time zone can exist but not be observed at the same time
- delecti 5y ago
- thezapxs 5y agoThe thing is, because the deletion of the BGP, they have ruled out the cyberattack but what happened doesn't make sense
- amachefe 5y agoFacebook.com is up now, looks like the issue now is the billions of request that is DDoS on the DNS servers
- tremon 5y agoWhy? Do you think there's billions of widely-used recursive resolvers in the world? Each resolver only needs to contact the Facebook DNS servers once per hostname, the end-user requests are all served from cache.
- cwilkes 5y agoUpdating BGP configs should go through a flowchart like this: Do you want to update BGP? No: exit Yes: type this random 100 character phrase to continue no copy paste
- toomuchtodo 5y ago“Are you sure? Can people you need to recover from this change already in the building?”
- _joel 5y agoMaybe a timed rollback with the previous state stored on the device that needs to be rolled back, althogh if you're doing this at facebook scale I'm sure that's a little more difficult than it sounds, perhaps.
- hinkley 5y agoLike the little dialog when you change your video settings.
- cyanydeez 5y agonot just in the building, but are there any microsoft enabled security doors between you and them
- rawoke083600 5y agoor like an iptables or Windows desktop screen resolution change :)
- alksjdalkj 5y agoQuestion about the WARP map - I assume the grey countries are places where Cloudflare doesn't have any presence, but what about Egypt/Oman? Why are they green? And why is Australia orange and not red?
- rswail 5y agoBecause most of the outage happened in the early hours of the morning AU time, so not that many people online.
- davidjytang 5y agoAlso curious about the green part.
- jgrahamc 5y agoIt's an anomaly. I think that was right on the edge of "not enough traffic to report on".
- jgrahamc 5y agoWe didn't have enough data from those locations.
- ormax3 5y agoEgypt has over 100 million people. Facebook/WhatsApp is quite popular in that part of the world, so I would image it had significant traffic..
- WaitWaitWha 5y agoThis provides a good set of details (mostly educational) what happened up to but not including the how and why the BGP routes were withdrawn (who sent the UPDATE packets to the neighboring ASes?). The most "natural" occurrence that I can think of is best-path change. If a "better" route between AS is added, the now-second-best routes are withdrawn. Correct me if I am wrong, but there is no way of determining the source of the withdraw message (UPDATE)...
- WaitWaitWha 5y agoLove to hear the reason for the downs.
- asadjb 5y agoThat info. would probably come from FB though, not Cloudflare. At least that's how I understand the incident.
- WaitWaitWha 5y ago> "Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. " and > "... its root cause was a faulty configuration change on our end." From https://engineering.fb.com/2021/10/04/networking-traffic/outage/ https://engineering.fb.com/2021/10/04/networking-traffic/out...
- robocat 5y agoAlso see Krebs’ article on the outage (although he hasn’t yet updated it after the problem was rectified): https://krebsonsecurity.com/2021/10/what-happened-to-facebook-instagram-whatsapp/ https://krebsonsecurity.com/2021/10/what-happened-to-faceboo...
- Ansil849 5y agoIs anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it. Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing glorified bikeshedding, and providing an explainer of how BGP works that can be gleaned by reading virtually any introductory text on it, and using it as a marketing opportunity.
- samhwr 5y agoIt’s massively viral news about the exact stuff they specialise in. I’m not too surprised that they’d be putting out content. After all, Facebook is hardly in a position to do so ;)
- Ansil849 5y agoIt's not surprising, no, but it is irritating because it's just crass marketing.
- missedthecue 5y agoI mean... they're in the business of keeping websites online. It's natural that they document these events both for their own research and to market their product.
- rjvir 5y agoIt's merely an informative blog post on a topic many people are interested in. I see nothing wrong with Cloudflare just explaining what happened, even though they had nothing to do with it.
- Ansil849 5y ago> I see nothing wrong with Cloudflare just explaining what happened, even though they had nothing to do with it. You kinda inadvertently highlighted the issue: because they had nothing to do with it, they do not know what actually happened. They can pontificate about likely causes, just like others in the industry can, but they have no idea what actually caused the issue.
- zanethomas 5y agoAll very amusing. But most amusing was that facebook employees' badges wouldn't work. Hilarious. Can't get in the office if facebook.com is not reachable. (or whatever the unreachable service was)
- unityByFreedom 5y agoAh, so no playing oculus either?
- 14 5y agoCurious about this as well could not find anything on initial searches but I honestly hope so. The one reason I have not grabbed the oculus is because the Facebook requirement.
- crmd 5y agoWhen I hear Facebook leadership talking about the VR vision, I’m reminded of IBM in the early 2000s encouraging employees to claim their second life avatars and setup cyber customer briefing centers.
- paul7986 5y agoExactly ... like I am going to strap on a headset and work with my co-workers memojis for 8 hours or even less. Laughable and VR has been around since the 90s with the same form factor .. it hasn't evolved as it will always be strap this bulky thing to your head & isolate yourself from the world. Maybe it will catch on for videogames but to co-work in a virtually reality world HA. AR Glasses at least their form factor has changed since Google Glass and will continue to evolve some. Google Glasses to how Facebook Stories sunglasses look which look like sunglasses billions are use to wearing and use daily. Billions will never strap on a headset for hours to co-work. Billions will adopt AR Glasses as it takes a familiar form factor/daily life product and enhances it like the iPHone enhance our daily lives.
- 5y ago
- pier25 5y agoThis is all very interesting, but why couldn't FB just push an update remotely to solve this? I've seen mentions everywhere that to fix this they required physical access to the BGP routers.
- ninkendo 5y agoHow would they push an update if they couldn’t reach their own network? My understanding is that if a bad BGP route was pushed and your whole network (typically accessed via VPN while the engineers WFH) is probably unreachable from any of you’re employees’ homes, it’s hard to get the right people on site to make a fix, or talk the remote hands folks at the date center through it over the phone. Troubleshooting is hard, troubleshooting while you have no access to your own network has got to be just that much harder.
- gukov 5y agoEveryone knows deploying anything critical on the first and the last day of a work week is not a good idea. Is Facebook run differently?
- lokar 5y agoThis and other stories seem to assume that a BGP issue is the root cause. That may be, but there are other possibilities. Some other internal issues could have broken many systems, including BGP. They could have had some sort of internal systems failure, and intentionally withdrew BGB to cut off the flood of connection attempts (making recovery easier). Many possibilities.
- ofou 5y agoA network intern it's saying oh-oh I mess up something...
- cottsak 5y agoamazing write ups from the CF guys
- nkssy 5y agoIs there a petition to make this permanent? Asking for a friend The article itself was a good exploration of the impact of BGP and what happens when network advertisements stop and the associated network disappears in a puff of global forgetfulness. Might be time to poke around BGP as well. A lab setup might be a good toy.
- nancyp 5y agoMy CCNA friends liked the thread. Oh wait she was on 60 minutes last night.
- paul7986 5y agoThis happens the day after the whistleblower's 60 Minutes interview? Maybe another rogue (whistleblower is surprised Facebook is a for profit business focused on profit .. not saying good or bad given how big facebook is but really how do you police the world and what side do you take in all the madness/negative parts that make up humanity thus facebook) Facebook employee took it offline ... any chance of that?
- quickthrower2 5y agoI'm on 1.1.1.1 and well..... FB was down.
- dboreham 5y agoThey used the "foot blunderbus".
- geocrasher 5y agoHow I explained this to friends and family: Imagine that the only way to get to Los Angeles is to use a GPS enabled device. All of the maps know how to get there from anywhere. But imagine that every GPS took Los Angeles off the map. Los Angeles is still there, but nobody knows how to get there. That's what happened to Facebook. We don't yet know why, however.
- js2 5y agoThose short TTLs that FB likely has on its DNS records probably bit it in the ass today. If it had longer TTLs, caching would have helped it. Curious though that recursive resolvers won't serve an expired cached record when they can't reach the authoritative server. (I know that unbound can be configured to do so, but not sure about others.)
- asadjb 5y agoI don't think so. The root cause (as I understand) was that FB stopped advertising it's IP address space to the world. Even if you had the IP addresses of the FB servers, you would not find any route to access them. The fact that DNS was also not resolving was a symptom of the DNS servers also being unavailable since they were part of the same IP address space that was unadvertised.
- deleted 5y ago[deleted]
- cblconfederate 5y agoMaybe FB should switch to a single IP address to avoid burdening the internet with their DNS requests.
- codedeadlock 5y agoHow can you explain yesterday's outage (Facebook, Instagram, WhatsApp) to your parents? You are feeling hungry and went to food court. The food court (open area) has a lot of options. You sit down in front of Domino's (Facebook), since you want to eat garlic bread. Now, you can't order from the counter directly. The waiter will come to your seat and ask for the order. You ordered garlic bread from the waiter, but the guy at Domino's counter went missing. Your order was not reaching to the chef in kitchen as Domino's counter guy was not present. This explains why Domino's (Facebook) ecosystem was down, but what about other vendors? They had nothing to do with Facebook. To understand this, we need to go back to our food court again. Now, there are a lot of hungry people sitting outside Domino. Since they were not getting answer from one waiter as why their food is not on their table, they started disturbing all the waiters. Due to this, majority of the waiters were trying to figure out where the Domino's counter guy went and other food joints (read websites) were not able to fulfil their own orders. So although only Domino's was down, it appeared as if whole Food Court (Internet) was facing issues. Counter Guy at Domino's - Facebook Nameservers Waiters - DNS Servers (Cloudflare, Google, Akamai)
- corobo 5y agoAnalogies serve to confuse half the audience and make the other half go "actually it's more like .."
- miked85 5y agoI think that this may be more confusing than the actual explanation.
- codedeadlock 5y agoThanks for the feedback. I will simplify it further!
- 5560675260 5y agoThere is no need to make it any more complex than "facebook, the company, messed up, now their properties are broken". An overly elaborated analogy just makes you sound condescending.
- adminscoffee 5y agoi've honestly been coding all day, i didn't notice until i got on here and say everyone talking about it
- PaulRobinson 5y agoIt's been 20+ years since I had reason to know the internals of BGP, but I still carry a slight nervousness whenever I think about it. In many ways, it's beautiful in that it allows Autonomous Systems to come together and independently make the Internet. However, I always felt it was brittle in that it relied every AS to have edge routers aware of the entire AS routing map, and so when those routers went down hard, when they came back up they were blank and needed to learn the Internet again. I noted back then that on some routers a single spoofed UDP packet to a router getting a table from its peers could cause it to stop doing that and start again. In a time when those updates could take hours... well... that kept me awake at night sometimes. Propagation of routes was global and error prone. Even today, you can see on the spy glass websites rogue AS routers advertising crazy routes either by accident or on purpose as a hostile act (to try and get traffic for a target network through itself). There are tens of thousands of engineers globally nursing and managing this stuff to keep the whole thing going. Like I said, it's been a while since I last looked closely, and I imagine multiple improvements have been made in this space since that time, but if there was ever a protocol that needs a long, cold hard look for a replacement, it's quite possibly BGP. It's also the one - and only - non-crypto situation where if somebody asked me if a blockchain could be really very useful, I'd say "probably, yeah".
- nroets 5y agoSo much of the internet is about translating addresses from one format to another: Human readable to IP: DNS Private IP to Public IP: NAT MAC to Private IP: ARP Public IP to AS: BGP With a block chain DNS it may be possible to replace all of them with a single protocol, but it may well turn out to be an academic exercise.
- grlass 5y agoNamecoin [1] was an early fork of bitcoin, that did blockchain based DNS, with the ideas being actively discussed by Satoshi Nakamoto [2]. That deals with DNS (though how much has it been used?), but what about the layers of the stack? [1] https://en.wikipedia.org/wiki/Namecoin https://en.wikipedia.org/wiki/Namecoin [2] https://bitcointalk.org/index.php?topic=1790.222 https://bitcointalk.org/index.php?topic=1790.222
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- edf13 5y agoDidn't we used to have long TTLs on DNS hostnames to help with this very reason?
- teh_klev 5y agoOnce upon a time this would have been the norm. If we were performing any DNS related maintenance we'd drop the TTL to 5 mins a day or so before the maintenance window was due to start. Once happy we'd not broken anything then we'd bump back to 21600 or whatever. But I guess the move fast/break all the things crowd no longer have any patience for this kind of thing.
- belter 5y agoLets assume somebody messed up their BGP config. Does anybody know if they use Juniper? Why they would not do the changes via "the commit confirmed" option? The idea is you do something, you messed up and are locked out, the system will revert to previous config after a few minutes by itself. The story of people physically accessing systems only makes sense, if this was a hack... =============== Commit Confirmed "Suppose that despite all your efforts to insure your new configuration is correct before you commit it, something is overlooked and when you commit, you are locked out of the router." "Rather than just a simple commit, you can make the candidate configuration active with a commit confirmed command. With this command, the router waits 10 minutes for a second commit. If it does not receive that confirming command within those 10 minutes, the router automatically does a rollback and commit so that the previous configuration becomes active again." https://www.networkworld.com/article/2345600/managing-a-junos-configuration.html https://www.networkworld.com/article/2345600/managing-a-juno... "Committing a Configuration" https://www.juniper.net/documentation/us/en/software/junos/cli/topics/topic-map/junos-configuration-commit.html https://www.juniper.net/documentation/us/en/software/junos/c... ===============
- afinlayson 5y agoI imagine Zuckerberg doing the same thing Taika Waititi's character "Antoine" did at the end of Free Guy while watching the 60 minutes video.
- alfalfasprout 5y agoStuff like this happens... but the real question is why FB had to send employees to physically go to datacenters to fix this. Sounds like their OOB management had a dependency on FB being online which seems like a bad design. If you're that paranoid about security issue revokable one time pads to a select set of employees.
- tpike01 5y agoHow can a single configuration change cascade so quickly on facebooks network was it foul play or simply human error?
- tpike01 5y agoWas the outage a configureration error by the admins or was it foul play?
- pinix 5y agoLet's say that when Facebook cannot announce their DNS prefixes, the DNS resolvers from other ASes cannot reach their DNS servers. But why? Does Facebook change the IP addresses of their DNS servers and other web endpoints constantly? Shoudn't Facebook IP addresses be cached by DNS resolvers?