15 ms·
Every device with FB app is now DDoSing recursive DNS resolvers
- walrus01 5y agoIf you're an ISP that is anything bigger than a mom-and-pop operation, you should have at least 3 or 4 geographically distributed anycast recursive resolvers. Recursive DNS is pretty easy to do for really large volumes on a $600 1U server. It's not like the days of 15 years ago...
- fmajid 5y agoAh, but what about all the marketing analytics ISPs deploy on their DNS servers so that if your browser ever looked up viagra.com, forever will it grace your web browsing retargeting ad units? /s
- protomyth 5y agoYeah, I now know every user with the FB app installed. Its just wild to watch the log of all the phones asking for facebook.com.
- hansel_der 5y ago... or with the facebook-sdk in any installed app. btw; going trou the (dns-)logs of ppl you personally know is somewhat rude
- protomyth 5y agoI'm the sysadmin looking at a number, and I don't identify people individually unless I have a witness and a direct order to do so based on the policy manual. I take that very seriously. I will not break people's trust. I was asked about how it was affecting us.
- hansel_der 5y agomy apologies then!
- deleted 5y ago[deleted]
- Ajedi32 5y agoIs this specifically an issue with the Facebook app? Or is it just a predicable consequence of DNS responses no longer being cached due to query failures for a site as popular as Facebook?
- treesknees 5y agoIt is certainly not specific to Facebook, but the scale at which Facebook is referenced across websites and apps is pretty unique (I can only think of a few key players like Google who would cause a similar load.) And to clarify a bit, the queries aren't "no longer being cached due to query failures", it's because their TTL expires and the resulting SERVFAIL from the next query (which fails) isn't cached at all.
- john37386 5y agoAt this point people hope it will just restart so that we can all resume a normal life. There can only be 2 options. It will be fix very soon or it will be a hell of a night with a lot of coffee.
- deleted 5y ago[deleted]
- earth2mars 5y agoIs it just the people trying to connect or the app itself keep polling and trying to send information from devices to Facebook servers continuously.
- earth2mars 5y agoIs it just the people trying to connect or the app itself keep polling and trying to send information from devices to Facebook servers continuously.
- junon 5y agoBoth.
- lrem 5y agoIsn't this "every device with an app using the FB SDK"?
- agilob 5y agoEvery app with facebook SDK used for whatever reason, like login or metrics or ads...
- Fordec 5y agoIs it just me, or every time there's an issue with facebook server side, the FB SDK just absolutely damages any software with it installed as a dependency. The thought put into failover states under the hood is very lacking. It speaks to an ethos of if developers aren't a working data pipeline to facebook services, them and their own products can go pound sand.
- Rd6n6 5y agoDdos is a strong way to put this. Are we talking malware that is sending thousands of requests per device, or a bug from a connectivity issue?
- Jtsummers 5y agoIt's still a DDoS, just not an attack. Slashdotting a site is a DDoS, but usually not intended as a deliberate attack. Now take every WhatsApp, Facebook, Instagram, and Facebook Messenger user, every app that uses FB for user authentication, every site that does the same, every app and site that serves FB ads, every app and site that uses FB for metrics, and we have an unintentional DDoS just waiting to happen.
- tantalor 5y agoThe word "attack" has multiple definitions, including "act harmfully on" e.g., a heart attack. It does not require intent or aggression.
- sschueller 5y agoIs there no response code for a DNS to say "I don't have what you want right now, come back later but wait at least xxx seconds"? I guess alternatively you could return garbage (127.0.0.1) with a 5 min ttl or so to get clients to backoff but also problematic.
- Computeiful 5y agoClients should really be using something like exponential backoff ethernet-style.
- jeffbee 5y agoJust imagine how annoying that would be. Your local resolver has a blip of connectivity and during the time it cannot contact an origin nameserver it burns all of the reasonable retry intervals. After a few seconds its link comes back up but the resolver is still waiting because of "backoff". Exponential backoff is a really bad antipattern that harms users. There are other, better ways to shed load. Also, there's no evidence that any DNS infrastructure was overloaded during this event, so what are we even discussing?
- ReactiveJelly 5y ago6 seconds of wait for a 3-second blip still sounds reasonable.
- PeterisP 5y agoNot really; if I unplug a cable, connections break; but if I plug it back in they can and should resume much faster than 6 seconds. And what about a five minute "blip"? Arguably it should resume proper operations as soon as the link goes back up, not some (unknown to user) time afterwards.
- Spivak 5y agoYes but event driven is way different. That should be instant because the software can be told by the system that connectivity was restored and respond immediately since it’s aware of the hardware being plugged in.
- xenonite 5y agoAnd even Hacker news is strangely slow to respond.
- reayn 5y agoyeah I thought I was the only one, never really noticed this on HN before...
- adtac 5y agoreally lol? HN goes down every full moon day or something like that
- bifrost 5y agoYep, with FB and IG down, people spending time on a much more important site.... This site.
- mrep 5y agoLots of people interested in the event and normal web response queries their sql database for user data like upvotes which is their limiter. Logout which I believe they just forced for all users sends you to the cache and it'll be fast. dang@ mentioned it during the giant S3 outage a few years ago.
- munk-a 5y agoAh nice - I noticed it just got a lot more snappy to respond.
- littlecranky67 5y agoI'd thank anybody who would post a tutorial/configfile to setup a DNS server (dnsmasq?), forcefully caching even failed requests to a configurable timeout, and large cache sizes. We might need them in case DNS servers going down under the load of requests from "smart devices" :)
- agilob 5y ago> forcefully caching even failed requests to a configurable timeout I've been doing ~SRE for 1.5 years and I've worked or helped on 3 outages related to negative DNS.. Please don't use negative cache, if you don't know how enough about DNS and can't monitor it
- littlecranky67 5y agoI wouldn't suggest any ISP should do that (and I am none) but probably host this for own personal usage/home networks. If recursive DNS servers go down under the load of "smart devices", having a local copy of a larger number/set of IPs I usually visit might come in handy (and none of my requests would worsen the issue of server overload).
- agilob 5y agoThis is 'cache', my OpenWRT router has this for thousands of records, but negative cache means: "remember this domain doesn't exist and don't retry asking other DNS providers". This is very dangerous. Your browser AND operating system AND router already provide DNS caching, it's not something average user should even think about. You might want to consider it when things in your ISP go wrong (hello BT), or majority of computer request the same domains frequently, but then again, your router should do it already.
- littlecranky67 5y agoWell now in retrospect, Cloudflare claims to have enabled SERVFAIL caching to mitigate, and other commenters here say their ISPs did similar. You might want to overthink the negative caching strategy.
- littlecranky67 5y agoOh great, and at some point when key ISP DNS servers crack under the load, more and more websites will appear to "go down" from a users perspective - suddenly gmail.com and outlook.com not working. More and more people reload websites, restart devices etc. and increase the load even further. People fall back to using SMS/telephone, but since it is not used to that heavy load of 2021, soon phone calls fail. With FB, WA, Email and Phone "down", engineers can't be reached to fix this. And if they can, they fail to call the Uber to get them somewhere. And even if they could, the streets are congested with people that cannot communicate remotely so try to get somewhere to convey messages in-person. Hope they will just go back to fix FB and this is just in my head :)
- IgorPartola 5y agoIsn’t there an easy fix to just add a bullshit record for FB to DNS until this blows over? I am also thinking that all the poorly coded sites that don’t work unless the Share on Facebook button loads are also going to hemorrhage money. So are all the e-commerce sites that rely on Login with FB. I hope this results in everyone rethinking adding all that shit to their infrastructure before the next time.
- noahtallen 5y agoMy understanding is that the Facebook network itself was unreachable from the internet because of BGP. So even if an IP was resolved from DNS, that IP wouldn’t get routed to Facebook because it withdrew its routes from its peers where it connects to ISPs via BGP
- AnimalMuppet 5y agoNot if the IP was 127.0.0.1...
- messe 5y agoThat's precisely what my ISP did. In fact, they went a step further and appear to have hijacked all unencrypted DNS queries. I was able to run DNS queries using 'example.com' as my DNS server and get a response.
- idatum 5y agoNever having explicitly queried fb.com before I never noticed how they (face:b00c) got clever with their IPv6 address: 2a03:2880:f1ff:83:face:b00c:0:25de
- ceejayoz 5y agoFacebook also used brute force to get facebookcorewwwi.onion on Tor a while back. https://en.wikipedia.org/wiki/Facebook_onion_address https://en.wikipedia.org/wiki/Facebook_onion_address
- ipaddr 5y agofacebookwkhpilnemxj7asaniu7vnjjbiltxjqhye3mhbshg7kx5tfyd.onion is the latest
- ReactiveJelly 5y agoIs it really brute force when they had to put a big list of keywords in and make a backronym for whatever combo they happened to find first?
- colinmhayes 5y agoYes? From what I read they had an entire datacenter work on this for multiple days.
- yjftsjthsd-h 5y agoThat's just brute forcing from both directions
- 0xdeadb00f 5y agoVanity onion domains are usually created via brute force, yes, but it isn't something looked down upon. Heaps of services do this.
- drzaiusapelord 5y agoI would like to know how much pollution was created for that vanity url.
- justahuman1 5y agoIs it possible for facebook to instead rely on an anycast IP rather than DNS for their (non-web) phone apps?
- bifrost 5y agoNo. But FB's DNS is anycasted. FB's eggs were all in one basket, and the basket broke.
- treesknees 5y agoYes and no. Yes they could technically hardcode an anycasted IP address, however it'd be less reliable. Also you'd run into issues with TLS certificates. It'd be very inflexible and would probably result in more outages. But even if they did hardcode an IP, the underlying infrastructure for Facebook was also down not just DNS resolution of facebook.com. So even if the FB app didn't need to resolve a hostname, it would still be broken.
- unanswered 5y agoISPs keep floating charging Netflix for their own customers' data traffic; by the same logic DNS operators should be charging Facebook.
- fermentation 5y agoExcellent opportunity to set up a DNS-by-mail service. Just send me a letter with the names you want and I’ll get back to you within 3 to 5 business days!
- quickthrower2 5y agoOr go back to host files.
- Jumb0 5y agoCould anyone explain this so people with no DNS knowledge could understand?
- Jumb0 5y agoCould anyone explain this for people with no DNS knowledge?
- ricardo81 5y agoBGP is like the mail service. DNS is a translation from human readable addresses to machine addresses. BGP determines how to find those addresses from your server to theirs.
- MrStonedOne 5y agoThe internet phone book to convert .coms to ip addresses only scales up to the load level of hte internet because results are cached at multiple layers. your browser asks your computer which asks your router which asks your isp which asks .com's dns servers which ask facebook's dns servers for facebook's ip address. each layer will cache the results so say, even if 100,000 people in seattle want to know facebook.com's ip address, only the 5 or isps who provide internet in seattle have to ask for facebook.com's ip address, so 100,000 requests, but only 5 actual requests. even the per-device and per-home cache is helpful, because 500 page loads in 15 minutes still only results in 1 actual dns request. Here's the issue: Failures aren't cached. so while 100,000 people in 1 second trying to get facebook's ip only resulted in 5 requests going to the core dns servers, now results in 100,000 requests in 1 second trying going to the core servers.
- yourad_io 5y ago> Failures aren't cached. i thought so too but according to Cloudflare's blog post, at least some of them do: > Consequently, 1.1.1.1, 8.8.8.8, and other major public DNS resolvers started issuing (and caching) SERVFAIL responses. https://blog.cloudflare.com/october-2021-facebook-outage/ https://blog.cloudflare.com/october-2021-facebook-outage/
- slg 5y agoThis technology problem is a good metaphor for Facebook overall as a company. There is nothing fundamentally wrong with having your app regularly polling for DNS records when it can't find them, but that can be an actively harmful approach when you are the size of Facebook. Being that size comes with a whole swath of extra responsibilities to ensure that your behavior doesn't end up harming society as a whole.
- jmalicki 5y agoThere is something inherently wrong with that - it's why exponential backoff exists.
- hansel_der 5y ago> Being that size comes with a whole swath of extra responsibilities No! "that size" changes over time, so you have to be responsible all the time. analogy: read up on the '-f' flag for ping
- EGreg 5y agoLooks like they moved fast... AND BROKE THINGS!
- a-dub 5y agothis does not make sense to me. ttls tend to be short lived, but as i understand expirys are typically measured in days for this exact reason. if the ttl expires, and the upstream cannot be contacted, retry every retry seconds, and in the meantime, if you have the records (every isp has/had records for fb before the outage) that have not expired, then serve them. why did the public user dns servers (isps/google/cloudflare/etc) drop the facebook records? dns is designed to be able to handle an unreachable server. (or was some mitigation for some sort of abuse added along the line somewhere that invalidates zones if they're associated with completely unroutable addresses?)
- hoppla 5y agoThe whole point of the doomsday machine is lost if you keep it a secret!
- elzbardico 5y agoSo, if Facebook becomes sentient we can’t shut it down because it will break the internet?