4 ms·
Facebook should have had a panic room. Operations teams normally have a special room with a secure connection for situations like this, so that production can
by jart 5y ago
Facebook should have had a panic room.
Operations teams normally have a special room with a secure connection for situations like this, so that production can be controlled in the event of bgp failure, nuclear war, etc. I could see physical presence being an issue if their bgp router depends on something like a crypto module in a locked cage, in which case there's always helicopters.
So if anything, Facebook's labor policies are about to become cooler.
- mike_d 5y agoI would be absolutely shocked if they didn't. The problem is when your networking core goes down, even if you get in via a backup DSL connection or something to the datacenter, you can't get from your jump host to anything else.
- jart 5y agoIt helps if your dsl line is is bridging at layer 2 in the osi model using rotated psks, so it won't be impacted by dns/bgp/auth/routing failures. That's why you need to put it in a panic room.
- samhw 5y agoYup, it's terrifying how much is ultimately, ultimately dependent on dongles and trust. I used to work at a company with a billion or so in a bank account (obviously a rather special type of account), which was ultimately authorised by three very trusted people who were given dongles.
- cyberpunk 5y agoWhat did the dongles do?
- deleted 5y ago[deleted]
- samhw 5y agoSorry, I should have been clearer - the dongles controlled access to that bank account. It was a bank account for banks to hold funds in. (Not our real capital reserves, but sort of like a current account / checking account for banks.) I was friends with one of those people, and I remember a major panic one time when 2 out of 3 dongles went missing. I'm not sure if we ever found out whether it was some kind of physical pen test, or an astonishingly well-planned heist which almost succeeded - or else a genuine, wildly improbable accident.
- sulam 5y agoThat model works great, until you need to ask for permission to go into the office, and the way to get permission is to use internal email and ticketing systems, which are also down.
- jart 5y agoOperations teams don't need permission from some apparatchik to enter the office when production goes down. If they can't get in, they drill.
- Sebb767 5y ago> nuclear war I think you need some convincing to keep your SREs on-site in case of a nuclear war ;)
- cyberpunk 5y agoHey, if I can take the kids and there’s food for a decade and a bunker I’m probably in ;)