8 ms·
Interesting side effects: >Now, here's the fun part. @Cloudflare runs a free DNS resolver, 1.1.1.1, and lots of people use it. So Facebook etc. are down... gue
by r721 5y ago
Interesting side effects:
>Now, here's the fun part. @Cloudflare runs a free DNS resolver, 1.1.1.1, and lots of people use it. So Facebook etc. are down... guess what happens? People keep retrying. Software keeps retrying. We get hit by a massive flood of DNS traffic asking for http://facebook.com http://facebook.com
https://twitter.com/jgrahamc/status/1445066136547217413 https://twitter.com/jgrahamc/status/1445066136547217413
>Our small non profit also sees a huge spike in DNS traffic. It’s really insane.
https://twitter.com/awlnx/status/1445072441886265355 https://twitter.com/awlnx/status/1445072441886265355
>This is frontend DNS stats from one of the smaller ISPs I operate. DNS traffic has almost doubled.
https://twitter.com/TheodoreBaschak/status/1445073229970763781 https://twitter.com/TheodoreBaschak/status/14450732299707637...
- el-salvador 5y agoAnother side effect: Two of our local mobile operators are experiencing issues with phone calls due to network overload. https://twitter.com/claroelsalvador/status/1445081933331959831 https://twitter.com/claroelsalvador/status/14450819333319598...
- paganel 5y agoThe same happened in Romania with two of our mobile operators immediately after FB&all went down.
- karencarits 5y agoPossibly in Norway too (internet though, not phone calls) https://www.nrk.no/nyheter/internett-trobbel-hos-telia-1.15677753 https://www.nrk.no/nyheter/internett-trobbel-hos-telia-1.156...
- synaesthesisx 5y agoBelieve it or not, there are places in the world where FB products (WhatsApp specifically) are used as the primary communication platform for most people.
- hansel_der 5y agooh the irony! can't use the phone network to place a call b/c of fb-errors clogging the pipe
- htrp 5y agoam i correct in interpreting this as almost equivalent to a DDoS attack
- curiousgal 5y agoI don't understand that logic, wouldn't people interacting with the website normally also generate the same amount if not more DNS requests?
- bt1a 5y agoThere's a lot of caching involved in the chain of requests that would alleviate this request volume if things were working.
- eklbt 5y agoMy best guess is that after n many attempts to access the provided IP, the local DNS cache deletes the entry causing a miss. Then the cycle continues.
- pwagland 5y agoNo, since the positive response will normally be cached for "some time" dependant on a number of factors. The negative response on the other hand often won't get cached, again, dependent on settings.
- keithnoizu 5y agoYep, I always make it a point to cache cache-misses in my code.
- ars 5y agoSo then when I'm on some kind of blocked WiFi and nothing resolves, and I switch to a properly working WiFi your code will continue to fail? It's not so simple to cache misses - you don't know if it's a real miss or some kind of error. For example if Facebook cached the miss, then even when they are back up nothing would connect.
- paledot 5y agoIt's remarkable the effect even short TTL caching can have given enough traffic. I recall once caching a value that was being accessed on every page load with a TTL of 1s resulting in a >99% reduction in query volume, and that's nowhere near Facebook/internet backbone scale.
- deleted 5y ago[deleted]
- tyingq 5y agoI wonder if any big DNS servers will artificially cache a long TTL NXDOMAIN response for FB to reduce their load. Done wrong, it would extend the FB outage longer.
- coldtea 5y ago>Done wrong, it would extend the FB outage longer. Let's hope it's done wrong.
- nonsince 5y agoThis outage also affected whatsapp, one of the most widely-used communication technologies in the world. It also almost caused me to be locked out of my apartment, were it not for random chance and the kindness of a stranger, but I’m glad that you can feel smugly superior about it
- rytis 5y agoJust out of curiosity, and obviously if you can disclose... How does FB availability affect your ability to enter your apartment?
- dotancohen 5y agoNot OP, but such ideas usually stem from a misunderstanding of root cause. Facebook inaccessibility likely exposed poor assumptions or other flaws in e.g. "smart" devices or workflows. Those poor assumptions or other flaws are likely what got OP locked out of his apartment when Facebook went down, not Facebook itself going down.
- tyingq 5y agoAn apartment-complex entry app with Facebook login integration seems possible to me. That would be direct root cause. https://developers.facebook.com/docs/facebook-login/ https://developers.facebook.com/docs/facebook-login/
- htrp 5y agoam i correct in interpreting this as almost equivalent to a DDoS attack on DNS providers?
- universenz 5y agoFurther to this, doesn't Chrome and Safari quietly auto-ping/reload pages that "fail to connect" if they're left open in a tab or browser?
- yawaworht1978 5y agoHow often do the apps try to reconnect? Does anyone know?
- vakabus 5y agoI've launched Wireshark monitoring DNS traffic of roughly 5 phones. I've collected 19.8k DNS packets so far. Out of that, 5.1k packets are flagged with REFUSED or SERVFAIL. If I am not mistaken, it means that 51% of DNS requests fail. Looking at queries for graph.instagram.com, it looks like there are roughly 20 attempts in a sequence before it gives up. All in all, this could probably explain doubling of the DNS traffic. But the sample is rather small, so take it with a grain of salt.
- dotancohen 5y ago5.1 / 19.8 is much closer to a quarter than to half. But your point is still just as poignant at ~25% as it is at 51%.
- profile53 5y agoI think those are DNS round trips. So 1 packet to request and 1 to respond. E.g. 9.9k total requests of which 5.1k fail.
- samhw 5y agoEquivalent how? In volume? In intention?
- ricardo81 5y agoHopefully they're not DNS ANY requests? <ducks> (CF decided not to honour them some years ago)
- mrkramer 5y ago>Our small non profit also sees a huge spike in DNS traffic. It’s really insane. It's not crazy; people are panicking over Facebook, Instagram and WhatsApp being down and they keep trying to connect to those services. I mean I would panic too if I were social media junky.
- nonsince 5y agoIt’s not just "social media junkies", a very pretentious phrase to use considering you’re writing it in a comment on a social network. Hundreds of thousands of apps use Facebook APIs, often in the background too (including FB's own apps).
- executesorder66 5y agoIs "alcoholic" a very pretentious word to use considering that the person saying it has a beer once a week?
- polote 5y agoAlmost same thing happened when Signal went down: https://news.ycombinator.com/item?id=25803010 https://news.ycombinator.com/item?id=25803010 Signal apps DDoS'ed their own server Second comment was saying there is no point using Signal if they are down during 2 days. Only a few hours for FB yet but curiously nobody is saying the same :)
- SlowRobotAhead 5y ago> Software keeps retrying. We get hit by a massive flood of DNS traffic asking for http://facebook.com http://facebook.com If you aren’t using exponential backoff algorithms for your reconnect scheme - you should be! I have a device in the field, only a few thousand total, but we saw issues when our shared cloud would go down and everyone hammered it to get back up.