3 ms·
There is already a nice writeup on the current incident from Cloudflare at https://blog.cloudflare.com/october-2021-facebook-outage/ https://blog.cloudflare.com
by sarosh 5y ago
There is already a nice writeup on the current incident from Cloudflare at https://blog.cloudflare.com/october-2021-facebook-outage/ https://blog.cloudflare.com/october-2021-facebook-outage/
They key observations:
"Due to Facebook stopping announcing their DNS prefix routes through BGP, our and everyone else's DNS resolvers had no way to connect to their nameservers. Consequently, 1.1.1.1, 8.8.8.8, and other major public DNS resolvers started issuing (and caching) SERVFAIL responses.
But that's not all. Now human behavior and application logic kicks in and causes another exponential effect. A tsunami of additional DNS traffic follows.
This happened in part because apps won't accept an error for an answer and start retrying, sometimes aggressively, and in part because end-users also won't take an error for an answer and start reloading the pages, or killing and relaunching their apps, sometimes also aggressively."
- spyspy 5y ago> apps won't accept an error for an answer and start retrying, sometimes aggressively I'm certainly guilt of this. Retries make the world go round, and round again. I've been given attitude by teams that own downstream services. Them: "Why are you retrying so aggressively?" Me: "Why is your service so damn flakey?"
- throw0101a 5y ago> Retries make the world go round, and round again. Depends on the rate I would think: * https://en.wikipedia.org/wiki/Exponential_backoff https://en.wikipedia.org/wiki/Exponential_backoff
- yesbabyyes 5y agoSurely they are upstream from you if they need to rate limit you? (And that sounds like you giving, rather than being given, attitude.)
- drewcoo 5y agoI don't reload often, but when I do, I do it rapidly and in anger. -some tester I know
- belter 5y agohttps://www.thousandeyes.com/blog/facebook-outage-analysis https://www.thousandeyes.com/blog/facebook-outage-analysis