6 ms·
I remember Facebook had a similar story when they botched their BGP update and couldn't even access the vault. If you have circular auth, you don't have anythin
by 1970-01-01 1y ago
I remember Facebook had a similar story when they botched their BGP update and couldn't even access the vault. If you have circular auth, you don't have anything when somebody breaks DNS.
- crote 1y agoWasn't there an issue where they required physical access to the data center to fix the network, which meant having to tap in with a keycard to get in, which didn't work because the keycard server was down, due to the network being down?
- lenerdenator 1y agoNot to speak for the other poster, but yes, they had people experiencing difficulties getting into the data centers to fix the problems. I remember seeing a meme for a cover of "Meta Data Center Simulator 2021" where hands were holding an angle grinder with rows of server racks in the background. "Meta Data Center Simulator 2021: As Real As It Gets (TM)"
- junon 1y agoYep. And their internal comms were on the same server if memory serves. They were also down.
- simplyluke 1y agoI was there at the time, for anyone outside of the core networking teams it was functionally a snow day. I had my manager's phone number, and basically established that everyone was in the same boat and went to the park. Core services teams had backup communication systems in place prior to that though. IIRC it was a private IRC on separate infra specifically for that type of scenario.
- junon 1y agoThanks for the correction, that sounds right. I thought I had remembered IRC but wasn't sure.
- prmoustache 1y agoI remember working for a company who insisted all teams had to usr whatever corp instant messaging/chat app but our sysadmin+network team maintained a jabber server + a bunch of core documentation synchronized on a vps in a totally different infrastructure just in case and sure enough there was that a day it came handy.
- DevelopingElk 1y agoAWS, for the ultimate backup, relies on a phone call bridge on the public phone network.
- gregw2 1y agoAh, but have they verified how far down the turtles go, and has that changed since they verified it? In the mid-2000s most of the conference call traffic started leaving copper T1s and going onto fiber and/or SIP switches managed by Level3, Global Crossing, Qwest, etc. Those companies combined over time into Century Link which was then rebranded Lumen. As of last October, Lumen is now starting to integrate more closely with AWS, managing their network with AWS's AI: https://convergedigest.com/lumen-expands-fiber-network-to-support-aws/ https://convergedigest.com/lumen-expands-fiber-network-to-su... "Oh what a tangled web we weave..."
- wbl 1y agoI once suggested at work that we list diesel distributors using payment infra not on on us near our datacenters.
- kylecazar 1y agoWishful thinking, but I hope an engineer somewhere got to ram a door down to fix a global outage. For the stories.
- bluGill 1y agoI assume they needed their own air supply because the automatic poison gas system was activating. Then they had to dodge lazers to get to the one button that would stop the nuclear missle launch. add a bunch of other poinless scifi and evil villan lair tropes in as well...
- donalhunt 1y agoMost datacenters are fairly boring to be honest. The most exciting thing likely to happen is some sheet metal ripping your hand open because you didn't wear gloves. Still have my "my other datacenter is made of razorblades and hate" sticker. \o/
- formerly_proven 1y agoThey do commonly have poisonous gas though.
- maaaaattttt 1y agoNot sure if you’re joking but a relatively small datacenter I’m familiar with has reduced oxygen in it to prevent fires. If you were to break in unannounced you would faint or maybe worse (?).
- bob778 1y agoNot quite - while you can reduce oxygen levels, they have to be kept within 4pp so at worst, will make you light headed. Many athletes train at the same levels though so it’s easy to overcome.
- UltraSane 1y agoYes for some insane reason facebook had EVERYTHING on a single network. The door access not working when you lose BGP routes is especially bad because normal door access systems cache access rules on the local door controllers and thus still work when they lose connectivity to the central server.
- avidphantasm 1y agoThis sounds similar to AWS services depending on DynamoDB, which sounds like what happened here. Even if under the hood parts of AWS depend on Dynamo, it should be a walled-off instance separate from Dynamo available via us-east-1.
- UltraSane 1y agoThere should be many more smaller instances with smaller blast radius.
- holowoodman 1y agoDepends. Some have a paranoid mode without caching, because then a physical attacker cannot snip a cable and then use a stolen keycard as easily or something. We had an audit force us to disable caching, which promptly went south at a power outage 2 months later where the electricians couldn't get into the switch room anymore. The door was easy to overcome, however, just a little fiddling with a credit card, no heroic hydraulic press story ;)
- bcrl 1y agoThat's similar to the total outage of all Rogers services in Canada back on July 7th 2022. It was compounded by the fact that the outage took out all Rogers cell phone service, making it impossible for Rogers employees to communicate with each other during the outage. A unified network means a unified failure mode. Thankfully none of my 10 Gbps wavelengths were impacted. Oh did I appreciate my aversion to >= layer 2 services in my transport network!
- YokoZar 1y agoThat's kind of a weird ops story, since SRE 101 for oncall is to not rely on the system you're oncall for to resolve outages in it. This means if you're oncall for communications of some kind, you must have some other independent means of reaching eachother (even if it's a competitor phone network)
- bcrl 1y agoThat is heavily contingent on the assumption that the dependencies between services are well documented and understood by the people building the systems.
- YokoZar 1y agoAre you asserting that Rogers employees needed documentation to know that Rogers Wireless runs on Rogers systems?
- bcrl 1y agoRogers is perhaps best described as a confederacy of independent acquisitions. In working with their sales team, I have had to tell them where there facilities are as the sales engineers don't always know about all of the assets that Rogers owns. There's also the insistence that Rogers employees should use Rogers services. Paying for every Rogers employee to have Bell cell phone would not sit well with their executives. That the risk assessments of the changes being made to the router configuration were incorrect also contributed to the outage.