6 ms·

Oh great, and at some point when key ISP DNS servers crack under the load, more and more websites will appear to "go down" from a users perspective - suddenly g

Oh great, and at some point when key ISP DNS servers crack under the load, more and more websites will appear to "go down" from a users perspective - suddenly gmail.com and outlook.com not working. More and more people reload websites, restart devices etc. and increase the load even further. People fall back to using SMS/telephone, but since it is not used to that heavy load of 2021, soon phone calls fail. With FB, WA, Email and Phone "down", engineers can't be reached to fix this. And if they can, they fail to call the Uber to get them somewhere. And even if they could, the streets are congested with people that cannot communicate remotely so try to get somewhere to convey messages in-person. Hope they will just go back to fix FB and this is just in my head :)

IgorPartola 5y ago
Isn’t there an easy fix to just add a bullshit record for FB to DNS until this blows over? I am also thinking that all the poorly coded sites that don’t work unless the Share on Facebook button loads are also going to hemorrhage money. So are all the e-commerce sites that rely on Login with FB. I hope this results in everyone rethinking adding all that shit to their infrastructure before the next time.
- noahtallen 5y ago
  My understanding is that the Facebook network itself was unreachable from the internet because of BGP. So even if an IP was resolved from DNS, that IP wouldn’t get routed to Facebook because it withdrew its routes from its peers where it connects to ISPs via BGP
  - AnimalMuppet 5y ago
    Not if the IP was 127.0.0.1...
    - messe 5y ago
      That's precisely what my ISP did. In fact, they went a step further and appear to have hijacked all unencrypted DNS queries. I was able to run DNS queries using 'example.com' as my DNS server and get a response.
- jbotz 5y ago
  The problem isn't (wasn't) facebook.com's A records, it's that the authoritative nameservers for facebook.com are (were) unreachable. In theory someone could change the NS records for facebook on the .com nameservers to point somewhere else and serve up a fake facebook.com domain, but... 1) those NS records have a 6 hour TTL, so it would take a while be effective, and 2) who has the authority to do that?
  - junon 5y ago
    I would assume DNS providers were on call with FB staff to manually trigger flushes across the system to prop the records ASAP. Of course this doesn't affect client TTLs but resolution failures generally aren't cached anyway.
    - yourad_io 5y ago
      Speaking of flushing, today I learned that Google public DNS has an open "flush" function: https://dns.google.com/cache https://dns.google.com/cache
      - celsoazevedo 5y ago
        Cloudflare: https://1.1.1.1/purge-cache/ https://1.1.1.1/purge-cache/ OpenDNS: https://cachecheck.opendns.com/ https://cachecheck.opendns.com/
    - hamilyon2 5y ago
      Dns providers are basically every ISP in existence, it would be impractical to communicate with by phone calls
  - DaiPlusPlus 5y ago
    > those NS records have a 6 hour TTL Why on earth would any system need a TTL beyond 1 hour? The longer the TTL the longer your outage period will be if a cache is poisoned.
    - 5y ago
littlecranky67 5y ago
I wrote this as a pure joke, but now that I learned that SERVFAIL is not cached on browsers, clients, intermediate DNS servers [0] etc. I am curiously wondering what will be going on. It is not only FB apps, it is basically every website request (that uses FB JS for ads, tracking, etc.) that triggers a DNS request, which will be forwarded 1:1 from the ISP's DNS to the null-routed FB Subnet. This should put orders of magnitude more load on resolving DNS servers than usual. [0]: https://serverfault.com/questions/479367/how-long-a-dns-timeout-is-cached-for https://serverfault.com/questions/479367/how-long-a-dns-time...
- bowaggoner 5y ago
  I enjoyed the comment. You might like my short story along the same lines from a few years ago: https://bowaggoner.com/writeups/robust.html https://bowaggoner.com/writeups/robust.html
  - pomian 5y ago
    Very fun little story. Thanks for writing it.
- yourad_io 5y ago
  > but now that I learned that SERVFAIL is not cached on [...] intermediate DNS servers [0] I thought so too, but at least some do cache according to Cloudflare's blog post: > Consequently, 1.1.1.1, 8.8.8.8, and other major public DNS resolvers started issuing (and caching) SERVFAIL responses. https://blog.cloudflare.com/october-2021-facebook-outage/ https://blog.cloudflare.com/october-2021-facebook-outage/
  - im3w1l 5y ago
    Facebook is so large that even caching the SERVFAIL with a TTL measured in seconds should cut down a lot of traffic.
  - daper 5y ago
    As I understand my favorite resolver - unbound - has separate "infra" cache that also caches RTT times, and "down" status of name servers: https://nlnetlabs.nl/documentation/unbound/unbound.conf/#infra-keep-probing https://nlnetlabs.nl/documentation/unbound/unbound.conf/#inf...
asdff 5y ago
If its any consolation, I find SMS and telephone to be remarkably robust in these situations. In college during football games, the campus population would swell to probably 200k people within a few square miles each with a cell phone in a pocket. 3g and LTE would be worthless. Campus area wifi would be worthless. The only thing that would work is shutting all that off your phone and resorting to SMS and calling people over EDGE, but it worked flawlessly even with all the people stressing out a handful of towers at once.
- deleted 5y ago
  [deleted]
- littlecranky67 5y ago
  Football game's are planned events, and the carriers plan capacity accordingly. I remember sms+cell phone going down on several music festivals with 20.000+ participants (especially when it ended at around midnight). Only some carriers supported those "sponteneous" gatherings in the middle of nowhere with mobile cell towers that would keep connectivity going - but low-cost carriers never did.
- mschuster91 5y ago
  German carrier O2 was/is notorious for offering somewhat decent-ish service in urban areas under normal conditions - but major events that happen to have lots of people moving around the city like fans congregating to a soccer game with public transport, political rallies or your average drunkard festival (=Oktoberfest)? Instant collapse...
- deleted 5y ago
  [deleted]
- ReactiveJelly 5y ago
  It would be cool to see Matrix's 100bps experiment allow full apps (with E2EE and other modern features, as opposed to SMS) to handle horrid network conditions. https://matrix.org/blog/2019/03/12/breaking-the-100-bps-barrier-with-matrix-meshsim-coap-proxy https://matrix.org/blog/2019/03/12/breaking-the-100-bps-barr... (I think this was posted already, and it's from 2019, but some people must have missed it - I only saw it this year)
tarsius 5y ago
For those you understand Swiss German: Mani Matter – I han es Zündhölzli azündt https://www.youtube.com/watch?v=PkGatIgXERI https://www.youtube.com/watch?v=PkGatIgXERI
- Tainnor 5y ago
  I lit a match / And it made a fire. / And for my cigarette / I wanted to take the fire from the match. / But the match slipped from my hand and landed on the rug. / And it almost made a hole in the rug. Well, you know what can happen / If you're not careful with fire. / And for the light on a cigarette / A rug feels rather too expensive. / And from the rug the fire, alas, / Might have spread to the whole house / And who knows what would have happened thereafter? There would have been a fire in the district / And the fire fighters would have had to come. / They would have honked in the streets / And unloaded their pipes. / And they would have sprayed fire / And it would have been in vain / And the whole city would have burned with nothing to protect it. And the people would have jumped around / Fearing for their possessions. / They would have thought somebody had started a fire. / They would have grabbed assault rifles. / Everyone would have shouted: "Whose fault is it?" / The whole country would have rioted. / And they would have shot at the ministers behind the lecterns. The UN would have become involved / And the UN enemies as well. / To preserve peace in Switzerland / Both would have come with tanks. / It would have spread, little by little, / To Europe, to Africa. / There would have been a World War and humans would be no more. I lit a match / And it made a fire. / And for my cigarette / I wanted to take the fire from the match. / But the match slipped from my hand and landed on the rug. / Thankfully, I picked it back up.
BiteCode_dev 5y ago
It's already the case. It's been some times since some websites randomly are hard to reach for me. Changing DNS helped, but the best move has been a VPN to a far away country.

IgorPartola 5y ago

Isn’t there an easy fix to just add a bullshit record for FB to DNS until this blows over? I am also thinking that all the poorly coded sites that don’t work unless the Share on Facebook button loads are also going to hemorrhage money. So are all the e-commerce sites that rely on Login with FB. I hope this results in everyone rethinking adding all that shit to their infrastructure before the next time.

noahtallen 5y ago
My understanding is that the Facebook network itself was unreachable from the internet because of BGP. So even if an IP was resolved from DNS, that IP wouldn’t get routed to Facebook because it withdrew its routes from its peers where it connects to ISPs via BGP
- AnimalMuppet 5y ago
  Not if the IP was 127.0.0.1...
  - messe 5y ago
    That's precisely what my ISP did. In fact, they went a step further and appear to have hijacked all unencrypted DNS queries. I was able to run DNS queries using 'example.com' as my DNS server and get a response.
jbotz 5y ago
The problem isn't (wasn't) facebook.com's A records, it's that the authoritative nameservers for facebook.com are (were) unreachable. In theory someone could change the NS records for facebook on the .com nameservers to point somewhere else and serve up a fake facebook.com domain, but... 1) those NS records have a 6 hour TTL, so it would take a while be effective, and 2) who has the authority to do that?
- junon 5y ago
  I would assume DNS providers were on call with FB staff to manually trigger flushes across the system to prop the records ASAP. Of course this doesn't affect client TTLs but resolution failures generally aren't cached anyway.
  - yourad_io 5y ago
    Speaking of flushing, today I learned that Google public DNS has an open "flush" function: https://dns.google.com/cache https://dns.google.com/cache
    - celsoazevedo 5y ago
      Cloudflare: https://1.1.1.1/purge-cache/ https://1.1.1.1/purge-cache/ OpenDNS: https://cachecheck.opendns.com/ https://cachecheck.opendns.com/
  - hamilyon2 5y ago
    Dns providers are basically every ISP in existence, it would be impractical to communicate with by phone calls
- DaiPlusPlus 5y ago
  > those NS records have a 6 hour TTL Why on earth would any system need a TTL beyond 1 hour? The longer the TTL the longer your outage period will be if a cache is poisoned.
  - 5y ago

littlecranky67 5y ago

I wrote this as a pure joke, but now that I learned that SERVFAIL is not cached on browsers, clients, intermediate DNS servers [0] etc. I am curiously wondering what will be going on. It is not only FB apps, it is basically every website request (that uses FB JS for ads, tracking, etc.) that triggers a DNS request, which will be forwarded 1:1 from the ISP's DNS to the null-routed FB Subnet. This should put orders of magnitude more load on resolving DNS servers than usual. [0]: https://serverfault.com/questions/479367/how-long-a-dns-timeout-is-cached-for https://serverfault.com/questions/479367/how-long-a-dns-time...

bowaggoner 5y ago
I enjoyed the comment. You might like my short story along the same lines from a few years ago: https://bowaggoner.com/writeups/robust.html https://bowaggoner.com/writeups/robust.html
- pomian 5y ago
  Very fun little story. Thanks for writing it.
yourad_io 5y ago
> but now that I learned that SERVFAIL is not cached on [...] intermediate DNS servers [0] I thought so too, but at least some do cache according to Cloudflare's blog post: > Consequently, 1.1.1.1, 8.8.8.8, and other major public DNS resolvers started issuing (and caching) SERVFAIL responses. https://blog.cloudflare.com/october-2021-facebook-outage/ https://blog.cloudflare.com/october-2021-facebook-outage/
- im3w1l 5y ago
  Facebook is so large that even caching the SERVFAIL with a TTL measured in seconds should cut down a lot of traffic.
- daper 5y ago
  As I understand my favorite resolver - unbound - has separate "infra" cache that also caches RTT times, and "down" status of name servers: https://nlnetlabs.nl/documentation/unbound/unbound.conf/#infra-keep-probing https://nlnetlabs.nl/documentation/unbound/unbound.conf/#inf...

asdff 5y ago

If its any consolation, I find SMS and telephone to be remarkably robust in these situations. In college during football games, the campus population would swell to probably 200k people within a few square miles each with a cell phone in a pocket. 3g and LTE would be worthless. Campus area wifi would be worthless. The only thing that would work is shutting all that off your phone and resorting to SMS and calling people over EDGE, but it worked flawlessly even with all the people stressing out a handful of towers at once.

deleted 5y ago
[deleted]
littlecranky67 5y ago
Football game's are planned events, and the carriers plan capacity accordingly. I remember sms+cell phone going down on several music festivals with 20.000+ participants (especially when it ended at around midnight). Only some carriers supported those "sponteneous" gatherings in the middle of nowhere with mobile cell towers that would keep connectivity going - but low-cost carriers never did.
mschuster91 5y ago
German carrier O2 was/is notorious for offering somewhat decent-ish service in urban areas under normal conditions - but major events that happen to have lots of people moving around the city like fans congregating to a soccer game with public transport, political rallies or your average drunkard festival (=Oktoberfest)? Instant collapse...
deleted 5y ago
[deleted]
ReactiveJelly 5y ago
It would be cool to see Matrix's 100bps experiment allow full apps (with E2EE and other modern features, as opposed to SMS) to handle horrid network conditions. https://matrix.org/blog/2019/03/12/breaking-the-100-bps-barrier-with-matrix-meshsim-coap-proxy https://matrix.org/blog/2019/03/12/breaking-the-100-bps-barr... (I think this was posted already, and it's from 2019, but some people must have missed it - I only saw it this year)

tarsius 5y ago

For those you understand Swiss German: Mani Matter – I han es Zündhölzli azündt https://www.youtube.com/watch?v=PkGatIgXERI https://www.youtube.com/watch?v=PkGatIgXERI

Tainnor 5y ago
I lit a match / And it made a fire. / And for my cigarette / I wanted to take the fire from the match. / But the match slipped from my hand and landed on the rug. / And it almost made a hole in the rug. Well, you know what can happen / If you're not careful with fire. / And for the light on a cigarette / A rug feels rather too expensive. / And from the rug the fire, alas, / Might have spread to the whole house / And who knows what would have happened thereafter? There would have been a fire in the district / And the fire fighters would have had to come. / They would have honked in the streets / And unloaded their pipes. / And they would have sprayed fire / And it would have been in vain / And the whole city would have burned with nothing to protect it. And the people would have jumped around / Fearing for their possessions. / They would have thought somebody had started a fire. / They would have grabbed assault rifles. / Everyone would have shouted: "Whose fault is it?" / The whole country would have rioted. / And they would have shot at the ministers behind the lecterns. The UN would have become involved / And the UN enemies as well. / To preserve peace in Switzerland / Both would have come with tanks. / It would have spread, little by little, / To Europe, to Africa. / There would have been a World War and humans would be no more. I lit a match / And it made a fire. / And for my cigarette / I wanted to take the fire from the match. / But the match slipped from my hand and landed on the rug. / Thankfully, I picked it back up.

BiteCode_dev 5y ago

It's already the case. It's been some times since some websites randomly are hard to reach for me. Changing DNS helped, but the best move has been a VPN to a far away country.