14 ms·
There's still no connectivity to Facebook's DNS servers: > traceroute a.ns.facebook.com traceroute to a.ns.facebook.com (129.134.30.12), 30 hops max,
by Animats 5y ago
There's still no connectivity to Facebook's DNS servers:
> traceroute a.ns.facebook.com
traceroute to a.ns.facebook.com (129.134.30.12), 30 hops max, 60 byte packets
1 dsldevice.attlocal.net (192.168.1.254) 0.484 ms 0.474 ms 0.422 ms
2 107-131-124-1.lightspeed.sntcca.sbcglobal.net (107.131.124.1) 1.592 ms 1.657 ms 1.607 ms
3 71.148.149.196 (71.148.149.196) 1.676 ms 1.697 ms 1.705 ms
4 12.242.105.110 (12.242.105.110) 11.446 ms 11.482 ms 11.328 ms
5 12.122.163.34 (12.122.163.34) 7.641 ms 7.668 ms 11.438 ms
6 cr83.sj2ca.ip.att.net (12.122.158.9) 4.025 ms 3.368 ms 3.394 ms
7 * * *
...
So they're hours into this outage and still haven't re-established connectivity to their own DNS servers.
- john37386 5y agoYeah the patch to fix BGP to reach the DNS is sent by email to @facebook.com. Ooops no DNS to resolve the MX records to send the patch to fix the BGP routers.
- yoelo 5y agoSeriously? Is that how it works?
- john37386 5y agoI don't know. I doubt. It's just funny to think that you need email to fix BGP, but DNS is down because of BGP. You need DNS to send email which needs BGP. It's a kind of chicken and egg problem but at a massive scale this time.
- boshomi 5y agoSheera Frenkel: Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors. https://twitter.com/sheeraf/status/1445099150316503057 https://twitter.com/sheeraf/status/1445099150316503057
- _joel 5y agoYou'd think they'd have worked that into their DR plans for a complete P1 outage of the domain/DNS, but perhaps not, or at least they didn't add removal of BGP announcements to the mix.
- weisk 5y agoNo, the backbone of the internet is not maintained with patches sent in emails.
- chiluk 5y agohttps://lkml.org/ https://lkml.org/
- chiluk 5y agoYou are very wrong about that https://lkml.org/ https://lkml.org/
- cbarrick 5y agoClearly you and the person you replied to are talking about very different things.
- chiluk 5y agoYou are very wrong about that ;) https://lkml.org/ https://lkml.org/
- _joel 5y agoI think the sub-comment is confusing the linux kernel with BGP.
- deleted 5y ago[deleted]
- cranekam 5y agoNo. A network like Facebook's is vast and complicated and managed by higher-level configuration systems, not people emailing patches around. If this issue is even to do with BGP it's much more likely the root of the problem is somewhere in this configuration system and that fixing it is compounded by some other issues that nobody foresaw. Huge events like this are always a perfect storm of several factors, any one or two of which would be a total noop alone.
- KuiN 5y agoThe Swiss cheese model of accidents. Occasionally the holes all align. https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model
- soneil 5y agoThe fun part of BGP is they apparently make a lot of use of it within their network, not just advertising routes externally. https://engineering.fb.com/2021/05/13/data-center-engineering/bgp/ https://engineering.fb.com/2021/05/13/data-center-engineerin... (and yes, fb.com resolves)
- bronlund 5y agoYeah that's some pretty hardcore A/B testing right there.
- kiernanmcgowan 5y agoMy suspicion is that since a lot of internal comms runs through the FB domain and since everyone is still WFH, then its probably a massive issue just to get people talking to each other to solve the problem.
- secondcoming 5y agoUnlikely, PagerDuty was invented for this kind of thing
- kiernanmcgowan 5y agoOh I'm sure everyone knows whats wrong, but how am I supposed to send an email, find a coworkers phone number, get the crisis team on video chat etc etc if all of those connections rely on the facebook domain existing?
- ralphm 5y agoHence the suggestion for PagerDuty. It handles all this, because responders set their notification methods (phone, SMS, e-mail, and app) in their profiles, so that when in trouble nobody has to ask those questions and just add a person as a responder to the incident.
- korethr 5y agoYes, but Facebook is not a small company. Could PagerDuty realistically handle the scale of notifications that would be required for Facebook's operations?
- Spooky23 5y agoSure. As long as you plan for disaster. The place where I worked had failure trees for every critical app and service. The goal for incident management was to triage and have an initial escalation for the right group within 15 minutes. When I left they were like 96% on target overall and 100% for infrastructure.
- suyash 5y agoSource (hacker group Anonymous) : https://twitter.com/YourAnonOne/status/1445082304393719818 https://twitter.com/YourAnonOne/status/1445082304393719818
- kossTKR 5y agoNYT tech reporter Sheera Frenkel gives us this update: >Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors. https://twitter.com/sheeraf/status/1445099150316503057 https://twitter.com/sheeraf/status/1445099150316503057
- adriancooney 5y agoGot a good chuckle imagining a fuming Zuckerberg not being allowed into his office, thinking the world is falling apart.
- evolve2k 5y agoCan’t get in to fix error
- winternett 5y agoHeck of a coincidence I must say... I can imagine this affects many other sites that use FB for authentication and tracking. If people pay proper attention to it, this is not just an average run of the mill "site outage", and instead of checking on or worrying about backups of my FB data (Thank goodness I can afford to lose it all), I'm making popcorn... Hopefully law makers all study up and pay close attention. What transpires next may prove to be very interesting.
- forgotpwd16 5y agoIndeed, what happened shows a good reason not to rely only on social log-in for various sites.
- mdtancsa 5y agoIts partially there. C and D are still not in the global tables according to routeviews ie. 185.89.219.12 is still not being advertised to anyone. My peers to them in Toronto have routes from them, but not sure how far they are supposed to go inside their network. (past hop 2 is them) % traceroute -q1 -I a.ns.facebook.com traceroute to a.ns.facebook.com (129.134.30.12), 64 hops max, 48 byte packets 1 torix-core1-10G (67.43.129.248) 0.133 ms 2 facebook-a.ip4.torontointernetxchange.net (206.108.35.2) 1.317 ms 3 157.240.43.214 (157.240.43.214) 1.209 ms 4 129.134.50.206 (129.134.50.206) 15.604 ms 5 129.134.98.134 (129.134.98.134) 21.716 ms 6 * 7 * % traceroute6 -q1 -I a.ns.facebook.com traceroute6 to a.ns.facebook.com (2a03:2880:f0fc:c:face:b00c:0:35) from 2607:f3e0:0:80::290, 64 hops max, 20 byte packets 1 toronto-torix-6 0.146 ms 2 facebook-a.ip6.torontointernetxchange.net 17.860 ms 3 2620:0:1cff:dead:beef::2154 9.237 ms 4 2620:0:1cff:dead:beef::d7c 16.721 ms 5 2620:0:1cff:dead:beef::3b4 17.067 ms 6 * 7 * 8 *
- boshomi 5y agoKevin Beaumont: »The Facebook outage has another major impact: lots of mobile apps constantly poll Facebook in the background = everybody is being slammed who runs large scale DNS, so knock on impacts elsewhere the long this goes on.« https://twitter.com/GossiTheDog/status/1445118907187175427 https://twitter.com/GossiTheDog/status/1445118907187175427
- mikefromhome 5y agodead beef sounds about right
- Twisol 5y agoOh my gosh, their IPv6 address contains "face:b00c"... > 2a03:2880:f0fc:c:face:b00c:0:35
- therein 5y agoBesides being fun and quirky, it is actually useful for their sysadmins as well as sysadmins at other orgs. Well at least it will in 2036, when IPv6 goes mainstream.
- deleted 5y ago[deleted]
- rootusrootus 5y agoI just got off a short pre-interview conversation with a manager at Instagram and he had to dial in with POTS. I got the impression that things are very broken internally.
- otikthecessna 5y agoI read that as POTUS at first and paused for a minute
- dividedbyzero 5y agoWhat is POTS?
- woofcat 5y agoPlain old telephone system. Aka a phone.
- tacker2000 5y agoPlain Old Telephone System
- terramex 5y agoPlain old telephone service https://en.wikipedia.org/wiki/Plain_old_telephone_service https://en.wikipedia.org/wiki/Plain_old_telephone_service
- wolverine876 5y agoThis person has a POTS line in their current location, and a modem, and the software stack to use it, and Instagram has POTS lines and modems and software that connect to their networks? Wow. How well do Instagram and their internal applications work over 56K?
- rescbr 5y agoThey could have dialed in by their own cell phone though
- 5y ago
- deleted 5y ago[deleted]
- lbruder 5y agoLooks like they misconfigured a web interface that they can't reach anymore now that they're off the net. "anyone have a Cisco console cable lying around?"
- CommieBobDole 5y agoThe only one they have is serial and the company's one usb-to-serial converter is missing.
- Edman274 5y agoThe voices, stories, announcements, photos, hopes and sorrows of millions, no, literally billions of people, and the promise that they may one day be seen and heard again now rests in the hands of Dave, the one guy who is closest to a Microcenter, owns his own car and knows how to beat the rush hour traffic and has the good sense to not forget to also buy an RS-232 cable, since those things tend to get finicky.
- ozfive 5y agoGreat visual!
- Animats 5y ago"facebook.com" is registered with "registrarsafe.com" as registrar. "registrarsafe.com" is unreachable because it's using Facebook's DNS servers and is probably a unit of Facebook. "registrarsafe.com" itself is registered with "registrarsafe.com". I'm not sure of all the implications of those circular dependencies, but it probably makes it harder to get things back up if the whole chain goes down. That's also probably why we're seeing the domain "facebook.com" for sale on domain sites. The registrar that would normally provide the ownership info is down. Anyway, until "a.ns.facebook.com" starts working again, Facebook is dead.
- robalfonso 5y agoThis is not completely accurate. The whole reason a registrar with domain abc.com can use ns1.abc.com is because glue records are established at the registry, this allows a bootstrap that keeps you in from a circular dependency. All that said it’s usually a bad idea, for someone as large as Facebook they should have nameservers across zones ie a.ns.fb.com b.ns.fb.org c.ns.fb.co Etc…
- john37386 5y agoThere is always a step which involve to email the domain when a domain update its information with the registrar. In this case, facebook.com and registrarsafe.com are managed by the same NS. You need these NS to query the MX to send that update approval by email and unblock the registrar update. Glue records are more for performance than to make that loop. I'm maybe missing something but, hopefully they won't need to send an email to fix this issue.
- jfrunyon 5y agoI have literally never once received an email to confirm a domain change. Perhaps the only exception is on a transfer to another registrar (though I can't recall that occurring, either). To be fair, we did have to get an email from eurid recently for a transfer auth code, but that was only because our registrar was not willing to provide. In any case, no, they will not need to send an email to fix this issue.
- alexvoda 5y agoCan someone explain why it is also down when trying to access it via Tor using its onion address: http://facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5tfyd.onion/ http://facebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5t... Or when trying ips directly: https://www.lifewire.com/what-is-the-ip-address-of-facebook-818152 https://www.lifewire.com/what-is-the-ip-address-of-facebook-... I would have expected a DNS issue to not affect either of these. I can understand the onionsite being down if facebook implemented it the way a thirdparty would (a proxy server accessing facebook.com) instead of actually having it integrated into its infrastructure as a first class citizen.
- gamacodre 5y agoThe issue here is that this outage was a result of all the routes into their data centers being cut off (seemingly from the inside). So knowing that one of the servers in there is at IP address "1.2.3.4" doesn't help, because no-one on the outside even knows how to send a packet to that server anymore.
- spiantino 5y agoYou can get through to a web server, but that web server uses DNS records or those routes to hit other services necessary to render the page. So the server you hit will also time out eventually and return a 500
- deleted 5y ago[deleted]
- KaiserPro 5y agorouting was down _everywhere_ so tor is getting a better experience than most people by getting a 500 error
- keithnoizu 5y agoDNS is back, looks like systems are still coming online.