5 ms·

For Facebook and WhatsApp it looks like a DNS issue, name resolution fails with SERVFAIL: $ dig facebook.com ; <<>> DiG 9.16.21 <<>> facebook.com

• john37386 5y ago
  Even the Name servers are not returning any values. That's bad. dig @8.8.8.8 +short facebook.com NS These are usually anycasted, meaning that 1 ip return in NS are in fact several servers spread in several regions. They are distributed to closer match through agreements with ISP with the BGP protocol. Very interesting, because it seems that it took 1 DNS entry misconfiguration to withdraw M$ worth of devices from the internet.
  • variant 5y ago
    BGP goof? https://twitter.com/g_bonfiglio/status/1445056923309649926?s=20 https://twitter.com/g_bonfiglio/status/1445056923309649926?s...
• simlevesque 5y ago
  Maybe they tried everything else before that. At first it was working but they couldn't serve responses: https://i.imgur.com/UaCtOiX.png https://i.imgur.com/UaCtOiX.png Notice the "2020"
  • rvnx 5y ago
    The servers struggle to reply a basic 5xx answer. Two possibilities: - the DNS services internally have issues (most likely, as this could explain the snowball effect) - it could be also a core storage issue and all their VMs are relying on it and so they don't want to block third-party websites and think it will last for a long time, so they prefer to answer nothing for now in the DNS (so it will fail instantly to the client, and drain the application/database servers so they can reboot with less load)
    • zarzavat 5y ago
      I was on a video call during the incident. The service was working but with super-low bandwidth for 30 minutes, then I got disconnected and every FB property went down suddenly. Seems more suggestive of someone pulling the plug than a DNS issue, although it could also be both.
• ctur 5y ago
  It isn't just DNS. If you happen to have cached entries, the site is returning errors as well.
  • Nextgrid 5y ago
    Presumably the DNS being down also wreaks havoc in their internal infrastructure as services can no longer resolve each other's names.
    • qeternity 5y ago
      Internal services using public dns records?
      • msbarnett 5y ago
        Probably not, but their external and internal DNS may share infrastructure that's at the root of the failure
        • qeternity 5y ago
          Yikes, seems like an easy redundancy split.
          • fragmede 5y ago
            It seems like an easy redundancy split, but imagine driving two cars down the freeway at the same time, because you got a flat tire in one, the other day. In order to actually be redundant you need to have two sets of infrastructure to serve, and then if the internal one goes down, the external one's basically useless when the internal resolution's down anyway. Capacity planning (because you're inside Facebook and can't pretend that all data-centers ever-where are connected via an infinitely fast network) becomes twice as much work. How you do updates for a couple thousand teams isn't trivial in the first place, now you have to cordon them off appropriately? I don't know what Facebook's DNS serving infrastructure looks like internally, but it's definitely more complicated than installing `unbound` on a couple of left-over servers.
• pul 5y ago
  Jep, also from other caches: https://www.nslookup.io/dns-records/facebook.com https://www.nslookup.io/dns-records/facebook.com
• jbverschoor 5y ago
  Same here on facebook.com , [api]whatsapp.com (instagram.com works)
• Hokusai 5y ago
  Is this related in any way to what happened to Slack recently in their DNS?
  • etc-hosts 5y ago
    No https://lists.dns-oarc.net/pipermail/dns-operations/2021-September/021340.html https://lists.dns-oarc.net/pipermail/dns-operations/2021-Sep...
  • skywhopper 5y ago
    So far the pattern isn't the same. Slack published a DNSSEC record that got cached and then deleted it, which broke clients that tried to validate DNSSEC for slack.com. But in this case, the records are just completely gone. As if "facebook.com", "instagram.com", et al just didn't exist.
• WillPostForFood 5y ago
  I'm seeing similar DNS errors for many non-Facebook sites.
  • Spare_account 5y ago
    Do you have some examples?
    • chilldill 5y ago
      aws.amazon.com is down as well
    • janmo 5y ago
      I am getting DNS fails for wikipedia
      • peter_retief 5y ago
        wfm
      • marbex7 5y ago
        Wikipedia wfm.
    • WillPostForFood 5y ago
      normashooting.com - but only when, like the parent poster, using Google's DNS servers. Just switched to Cloudflare and it works. Using Google DNS: nslookup > normashooting.com Server: 8.8.8.8 Address: 8.8.8.8#53 * server can't find normashooting.com: SERVFAIL Using Cloudflare DNS servers: > normashooting.com Server: 1.1.1.1 Address: 1.1.1.1#53 Non-authoritative answer: Name: normashooting.com Address: 104.22.56.165 Name: normashooting.com Address: 104.22.57.165 Name: normashooting.com Address: 172.67.43.70
    • chilldill 5y ago
      cant login to aws console either
  • rstupek 5y ago
• hulitu 5y ago
  Thank god we have DoH.
  • dvratil 5y ago
    It's DNS over HTTPS. It relies on the same system as plain DNS, so DoH won't really help in this case...
• hikerclimber1 5y ago
  Everything is subjective. Especially laws.
• r721 5y ago
  John Graham-Cumming: >Between 15:50 UTC and 15:52 UTC Facebook and related properties disappeared from the Internet in a flurry of BGP updates. This is what it looked like to @Cloudflare. https://twitter.com/jgrahamc/status/1445065270272434176 https://twitter.com/jgrahamc/status/1445065270272434176 (thread) UPD >About five minutes before Facebook's DNS stopped working we saw a large number of BGP changes (mostly route withdrawals) for Facebook's ASN. https://twitter.com/jgrahamc/status/1445068309288951820 https://twitter.com/jgrahamc/status/1445068309288951820
• LordRishav 5y ago
  It's always DNS
  • sysadmindotfail 5y ago
    >It's always DNS How is this not the top comment? Underrated
• Animats 5y ago
  Even Google's 8.8.8.8 DNS server says can't find, SERVFAIL.