3 ms·
> since engineers got locked out of their systems Sounds like the same happened here: "Due to this withdrawal, Cloudflare engineers experienced added difficul
by mlyle 4y ago
> since engineers got locked out of their systems
Sounds like the same happened here:
"Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. We have backup procedures for handling such an event and used them to take control of the affected locations."
But Cloudflare had sufficient backup connectivity to fix it. I'm curious how Cloudflare does that today-- the solution long ago was always a modem on an auxiliary port.
- Melatonic 4y agoWorst case if I was designing this I would probably have a satellite connection running over Iridium at each of their biggest DC's Also lets face it - the utility of a trusted security guard/staff with an old fashioned physical key is pretty hard to screw up!
- jve 4y ago> the solution long ago was always a modem on an auxiliary port Now you can use mobile Internet (4G/5G)
- ccakes 4y agoCell coverage inside datacenters isn't always suitable, occasionally even by-design.
- cwt137 4y agoThey have their machines also connected to another AS, so when their network doesn't/can't route, they can still get to their machines to fix stuff.
- nijave 4y agoNot sure how common it is, but you can get serial OOBM devices accessible over cellular which would then give you access to your equipment. I'm surprised more places don't implement a "click here to confirm changes or it'll be rolled back in 5 minutes" like all those monitor settings dialogues