7 ms·
How on earth does EVERYTHING go down like this? It just seems unfathomable to me. Where is the single point of failure?
by grensley 6y ago
How on earth does EVERYTHING go down like this? It just seems unfathomable to me. Where is the single point of failure?
- notretarded 6y agoAkamai
- chaos_emergent 6y agoProbably an intern doing something dumb.
- mateo411 6y agoThe dumb thing would be giving the intern permission to bring down the whole site.
- vesinisa 6y agoIf an intern can bring down the whole presence of your company online it's not the intern who is dumb.
- jimmy2020 6y agobingo
- abhishekbasu 6y agosome startups would like to disagree
- dividedbyzero 6y agoWhile that's perfectly true, I'm sure a smart and determined intern could cause an astonishing amount of damage in many many orgs. Hopefully not at Facebook though.
- sleepydog 6y agoThe difference is whether or not it was intentional. If someone can bring everything down accidentally, the system is broken. Ditto if they can bring it down intentionally and cover their tracks so no one finds out who did it.
- MattGaiser 6y agoAn intern shouldn't have those permissions.
- jscheel 6y agoDNS (not necessarily this case, but man it can bring things down fast)
- cure 6y agoProbably DNS. It's always DNS ;)
- jonny_eh 6y agoor BGP, or auth.
- rogerdickey 6y agoHow does it stay up as much as it does :)
- fishnchips 6y agoBlood, sweat, tears and caffeine.
- kazen44 6y agoalso, some arcane shell script written by a retired engineer somewhere in the early start of the company.
- korethr 6y ago"And if these people stop, the world burns. Most people don't even know what sysadmins do, but trust me, if they all took a lunch break at the same time they wouldn't make it to the deli before you ran out of bullets protecting your canned goods from roving bands of mutants."[1] 1. https://www.stilldrinking.org/programming-sucks https://www.stilldrinking.org/programming-sucks
- lovedswain 6y agoThe global outage drinking game: bad DNS config push - 1 shot routing loop - 2 shots third party advertising your routes - 3 shots power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots Wolves ate through fiber - 5 shots And it was a full moon - 6 shots Single service failure, but service has not been restarted in 5 years, and no longer restarts in any documented fashion - 7 shots And service developers left the company to found a startup - 8 shots Expired internal SSL certificate - 9 shots Daylight savings changeover-induced database corruption - 10 shots Windows Update - 11 shots
- ASalazarMX 6y ago> Wolves ate through fiber - 5 shots I hope it's wolves, that would create a new category in https://cybersquirrel1.com/ https://cybersquirrel1.com/
- partcyborg 6y agoYou forgot: Hunter shot your fiber relay with a hunting rifle - 6 shots
- nabla9 6y agoApple decided to shut down Facebook.
- godelmachine 6y agoDST can be such a hassle. It affects our reports and change requests.
- nonameiguess 6y agoLarge scale network partitioning can happen. Data center ingress/egress can go down. If you shard to multiple data centers, it's still possible for an entire autonomous system to have an interconnect with a different AS break. That is rare as hell with the actual Internet because of all the route redundancy even between ASes, but I've seen it happen on more specialized networks. For a few days last year, the entire western half of the US was cutoff from the eastern half on Raytheon's internal TS network because someone digging a hole somewhere accidentally cut the wrong fiber line.
- tyingq 6y agoI'll be curious about it coming back up. It's the sort of site where there's an army of people continually trying to reload the page if something isn't working.