3 ms·

DNS of whatsapp.com is down. The app works fine

kro 3y ago
dig +trace reveals that only some of the nameservers don't respond (on IPv4, but got a fine response after trying an IPv6). ;; communications error to 66.111.50.12#53: timed out ;; communications error to 66.111.50.12#53: timed out ;; communications error to 66.111.50.12#53: timed out ;; communications error to 66.111.48.12#53: timed out whatsapp.com. 60 IN A 157.240.223.60 ;; Received 57 bytes from 2620:13e:100d:c::35#53(d.ns.whatsapp.net) in 119 ms Especially a.ns.whatsapp.net still seems down (IPv6 also). I wonder whether this could be a BGP thing again, it was modified very recently according to [1] but I don't understand too much about that. [1] https://lg.twelve99.net/?type=bgp&router=bei-b2&address=66.111.48.12 https://lg.twelve99.net/?type=bgp&router=bei-b2&address=66.1...
- toast0 3y ago
  I didn't get the sense of a BGP error, mtr to ipv4 [a-d].ns.whatsapp.net looked ok enough during the outage. I even got back responses from d.ns.whatsapp.net to mtr probes while not getting DNS responses. Although, now that DNS is working, the mtrs are pretty different, I'm not used to getting such wide geographical response.
paulddraper 3y ago
It's always DNS.
XCSme 3y ago
So is the conclusions: use IPs? Or how can you protect your app against those outages?
- toast0 3y ago
  Disclaimer: I worked at WhatsApp until 2019. We built in resilience against DNS failing because there's a lot of networks in the world, and some of them don't have consistently working DNS. Turns out it also helps a lot when someone hijacks your domain account (grr), or DDoSes your DNS provider (on my birthday too, grr), or you mess up DNS somehow (nobody was willing to tell me what happened yesterday, grr). Doesn't help much with BGP based outages though. The trick is if you bake IPs into your client packages, you need to have most or at least some of those IPs still working until the packages expire, and you need to have a process for server teams and client teams to update the lists for future releases as well as for server teams to know what's in the client builds in the wild. That's not too hard if you're running in bare metal hosting and your hosts are long lived. It's harder if you're running ephemeral cloud stuff (but there's ways to reserve IPs, so it's not _that_ hard). There's some additional difficulty if you're only running ipv4 servers and some of your clients only have NAT64 access to ipv4. But, it's 2023, you can and probably should make ipv6 work on your servers.

kro 3y ago

dig +trace reveals that only some of the nameservers don't respond (on IPv4, but got a fine response after trying an IPv6). ;; communications error to 66.111.50.12#53: timed out ;; communications error to 66.111.50.12#53: timed out ;; communications error to 66.111.50.12#53: timed out ;; communications error to 66.111.48.12#53: timed out whatsapp.com. 60 IN A 157.240.223.60 ;; Received 57 bytes from 2620:13e:100d:c::35#53(d.ns.whatsapp.net) in 119 ms Especially a.ns.whatsapp.net still seems down (IPv6 also). I wonder whether this could be a BGP thing again, it was modified very recently according to [1] but I don't understand too much about that. [1] https://lg.twelve99.net/?type=bgp&router=bei-b2&address=66.111.48.12 https://lg.twelve99.net/?type=bgp&router=bei-b2&address=66.1...

toast0 3y ago
I didn't get the sense of a BGP error, mtr to ipv4 [a-d].ns.whatsapp.net looked ok enough during the outage. I even got back responses from d.ns.whatsapp.net to mtr probes while not getting DNS responses. Although, now that DNS is working, the mtrs are pretty different, I'm not used to getting such wide geographical response.

paulddraper 3y ago

It's always DNS.

XCSme 3y ago

So is the conclusions: use IPs? Or how can you protect your app against those outages?

toast0 3y ago
Disclaimer: I worked at WhatsApp until 2019. We built in resilience against DNS failing because there's a lot of networks in the world, and some of them don't have consistently working DNS. Turns out it also helps a lot when someone hijacks your domain account (grr), or DDoSes your DNS provider (on my birthday too, grr), or you mess up DNS somehow (nobody was willing to tell me what happened yesterday, grr). Doesn't help much with BGP based outages though. The trick is if you bake IPs into your client packages, you need to have most or at least some of those IPs still working until the packages expire, and you need to have a process for server teams and client teams to update the lists for future releases as well as for server teams to know what's in the client builds in the wild. That's not too hard if you're running in bare metal hosting and your hosts are long lived. It's harder if you're running ephemeral cloud stuff (but there's ways to reserve IPs, so it's not _that_ hard). There's some additional difficulty if you're only running ipv4 servers and some of your clients only have NAT64 access to ipv4. But, it's 2023, you can and probably should make ipv6 work on your servers.