5 ms·
Wasn't there an issue where they required physical access to the data center to fix the network, which meant having to tap in with a keycard to get in, which di
by crote 1y ago
Wasn't there an issue where they required physical access to the data center to fix the network, which meant having to tap in with a keycard to get in, which didn't work because the keycard server was down, due to the network being down?
- lenerdenator 1y agoNot to speak for the other poster, but yes, they had people experiencing difficulties getting into the data centers to fix the problems. I remember seeing a meme for a cover of "Meta Data Center Simulator 2021" where hands were holding an angle grinder with rows of server racks in the background. "Meta Data Center Simulator 2021: As Real As It Gets (TM)"
- junon 1y agoYep. And their internal comms were on the same server if memory serves. They were also down.
- simplyluke 1y agoI was there at the time, for anyone outside of the core networking teams it was functionally a snow day. I had my manager's phone number, and basically established that everyone was in the same boat and went to the park. Core services teams had backup communication systems in place prior to that though. IIRC it was a private IRC on separate infra specifically for that type of scenario.
- junon 1y agoThanks for the correction, that sounds right. I thought I had remembered IRC but wasn't sure.
- prmoustache 1y agoI remember working for a company who insisted all teams had to usr whatever corp instant messaging/chat app but our sysadmin+network team maintained a jabber server + a bunch of core documentation synchronized on a vps in a totally different infrastructure just in case and sure enough there was that a day it came handy.
- DevelopingElk 1y agoAWS, for the ultimate backup, relies on a phone call bridge on the public phone network.
- gregw2 1y agoAh, but have they verified how far down the turtles go, and has that changed since they verified it? In the mid-2000s most of the conference call traffic started leaving copper T1s and going onto fiber and/or SIP switches managed by Level3, Global Crossing, Qwest, etc. Those companies combined over time into Century Link which was then rebranded Lumen. As of last October, Lumen is now starting to integrate more closely with AWS, managing their network with AWS's AI: https://convergedigest.com/lumen-expands-fiber-network-to-support-aws/ https://convergedigest.com/lumen-expands-fiber-network-to-su... "Oh what a tangled web we weave..."
- wbl 1y agoI once suggested at work that we list diesel distributors using payment infra not on on us near our datacenters.
- kylecazar 1y agoWishful thinking, but I hope an engineer somewhere got to ram a door down to fix a global outage. For the stories.
- bluGill 1y agoI assume they needed their own air supply because the automatic poison gas system was activating. Then they had to dodge lazers to get to the one button that would stop the nuclear missle launch. add a bunch of other poinless scifi and evil villan lair tropes in as well...
- donalhunt 1y agoMost datacenters are fairly boring to be honest. The most exciting thing likely to happen is some sheet metal ripping your hand open because you didn't wear gloves. Still have my "my other datacenter is made of razorblades and hate" sticker. \o/
- formerly_proven 1y agoThey do commonly have poisonous gas though.
- maaaaattttt 1y agoNot sure if you’re joking but a relatively small datacenter I’m familiar with has reduced oxygen in it to prevent fires. If you were to break in unannounced you would faint or maybe worse (?).
- bob778 1y agoNot quite - while you can reduce oxygen levels, they have to be kept within 4pp so at worst, will make you light headed. Many athletes train at the same levels though so it’s easy to overcome.
- UltraSane 1y agoYes for some insane reason facebook had EVERYTHING on a single network. The door access not working when you lose BGP routes is especially bad because normal door access systems cache access rules on the local door controllers and thus still work when they lose connectivity to the central server.
- avidphantasm 1y agoThis sounds similar to AWS services depending on DynamoDB, which sounds like what happened here. Even if under the hood parts of AWS depend on Dynamo, it should be a walled-off instance separate from Dynamo available via us-east-1.
- UltraSane 1y agoThere should be many more smaller instances with smaller blast radius.
- holowoodman 1y agoDepends. Some have a paranoid mode without caching, because then a physical attacker cannot snip a cable and then use a stolen keycard as easily or something. We had an audit force us to disable caching, which promptly went south at a power outage 2 months later where the electricians couldn't get into the switch room anymore. The door was easy to overcome, however, just a little fiddling with a credit card, no heroic hydraulic press story ;)