5 ms·

I can't fathom how they didn't plan for this. In any business of size, you have to change configuration remotely on a regular basis, and can easily lock yoursel

I can't fathom how they didn't plan for this. In any business of size, you have to change configuration remotely on a regular basis, and can easily lock yourself out on a regular basis. Every single system has a local user with a random password that we can hand out for just this kind of circumstance...

chasd00 5y ago
Another Monday morning at a boring datacenter job, i bet they weren't even there yet at 830 when the phones started ringing.
- bink 5y ago
  You mean the VOIP phones that could no longer receive incoming calls?
- mro_name 5y ago
  phones? how lame.
  - radomir_cernoch 5y ago
    It certainly wasn't the Messenger.
  - pmlnr 5y ago
    Phones - the old, analogue, direct cable ones - were self-sustaining, and kept running even when there was a power cut in the house.
    - mro_name 5y ago
      yes, indeed. Reliability. That's so 20th century. #lame. (Actually not lame at all in my eyes)
- steelframe 5y ago
  Assuming anyone can actually look up the phone numbers to call.
- CydeWeys 5y ago
  There should be 24/7 on-site rotations. I wonder if physical presence was cut on account of COVID?
shadowgovt 5y ago
Organizational complexity grows super-linearly; in general, the number of people a company can hire per unit time is either constant or grows linearly. Google once had a very quiet big emergency that was, ironically(1), initiated by one of their internal disaster-recovery tests. There's a giant high-security database containing the 'keys to the kingdom', as it were... Passwords, salts, etc. that cannot be represented as one-time pads and therefore are potentially dangerous magic numbers for folks to know. During disaster recovery once, they attempted to confirm that if the system had an outage, it would self-recover. It did not. This tripped a very quiet panic at Google because while the company would tick along fine for awhile without access to the master password database, systems would, one by one, fail out if people couldn't get to the passwords that had to be occasionally hand-entered to keep them running. So a cross-continent panic ensued because restarting the database required access to two keycards for NORAD-style simultaneous activation. One was in an executive's wallet who was on vacation, and they had to be flown back to the datacenter to plug it in. The other one was stored in a safe built into the floor of a datacenter, and the combination to that safe was... In the password database. They hired a local safecracker to drill it open, fetched the keycard, double-keyed the initialization machines to reboot the database, and the outside world was none the wiser. (1) I say "ironically," but the actual point of their self-testing is to cause these kinds of disruptions before chance does. They aren't generally supposed to cause user-facing disruption; sometimes they do. Management frowns on disruption in general, but when it's due to disaster recovery testing, they attach to that frown the grain of salt that "Because this failure-mode existed, it would have occurred eventually if it didn't occur today."
- hnaccy 5y ago
  what if the executive had been pick-pocketed
  - shadowgovt 5y ago
    EDIT: I had mis-remembered this part of the story. ;) What was stored in the executive's brain was the combination to a second floor safe in another datacenter that held one of the two necessary activation cards. Whether they were able to pass it to the datacenter over a secure / semi-secure line or flew back to hand-deliver the combination I do not remember. If you mean "Would the pick-pocket have access to valuable Google data," I think the answer is "No, they still don't have the key in the safe on the other continent." If you mean "Would the pick-pocket have created a critical outage at Google that would have required intense amounts of labor to recover from," I don't know because I don't know how many layers of redundancy their recovery protocols had for that outage. It's possible Google came within a hair's breadth of "Thaw out the password database from offline storage, rebuild what can be rebuilt by hand, and inform a smaller subset of the company that some passwords are now just gone and they'll have to recover on their own" territory.
outworlder 5y ago
> I can't fathom how they didn't plan for this Maybe because they were planning for a million other possible things to go wrong, likely with higher probability than this. And busy with each day's pressing matters.
- deleted 5y ago
  [deleted]
- mynameisvlad 5y ago
  A config change gone bad? That’s like failure scenarios 101. That should be the second on the list, after “code change gone bad”.
  - jfrunyon 5y ago
    Exactly! Obviously they have extremely robust testing and error catching on things like code deploys: how many times do you think they deploy new code a day? And at least personally, their error rate is somewhere below 1%. Clearly something about their networking infrastructure is not as robust.
  - pupdogg 5y ago
    Right? Especially on global scale. Something doesn't add up!
    - TheOtherHobbes 5y ago
      Curious/unfortunate timing. The day after a whistleblower docu and with a long list of other legal challenges and issues incoming.
- smrtinsert 5y ago
  Haha sure. They were too busy implementing php compilers to figure out that "whole DR DNS thing" rotflmao. I'd remove Facebook from my resume.
- jfrunyon 5y ago
amalcon 5y ago
Most likely they did plan for this. Then, something happened that the failsafe couldn't handle. E.g. if something overwrites /etc/passwd, having a local user won't help. I'm not saying that specific thing happened here -- it's actually vanishingly unlikely -- but your plan can't cover every contingency.
- deleted 5y ago
  [deleted]
- robalfonso 5y ago
  Agreed, it’s also worth mentioning that at the end of every cloud is real physical hardware, and that is decidedly less flexible than cloud, if you locked yourself out of a physical switch or router you have many fewer options.
- onecommentman 5y ago
  In risk management cultures where consequences from failures are much, much higher, the saying goes that “failsafe systems fail by failing to be failsafe”. Explicit accounting for scenarios where the failsafe fails is a requirement. Great truths of the 1960s to be relearned, I guess.

chasd00 5y ago

Another Monday morning at a boring datacenter job, i bet they weren't even there yet at 830 when the phones started ringing.

bink 5y ago
You mean the VOIP phones that could no longer receive incoming calls?
mro_name 5y ago
phones? how lame.
- radomir_cernoch 5y ago
  It certainly wasn't the Messenger.
- pmlnr 5y ago
  Phones - the old, analogue, direct cable ones - were self-sustaining, and kept running even when there was a power cut in the house.
  - mro_name 5y ago
    yes, indeed. Reliability. That's so 20th century. #lame. (Actually not lame at all in my eyes)
steelframe 5y ago
Assuming anyone can actually look up the phone numbers to call.
CydeWeys 5y ago
There should be 24/7 on-site rotations. I wonder if physical presence was cut on account of COVID?

shadowgovt 5y ago

Organizational complexity grows super-linearly; in general, the number of people a company can hire per unit time is either constant or grows linearly. Google once had a very quiet big emergency that was, ironically(1), initiated by one of their internal disaster-recovery tests. There's a giant high-security database containing the 'keys to the kingdom', as it were... Passwords, salts, etc. that cannot be represented as one-time pads and therefore are potentially dangerous magic numbers for folks to know. During disaster recovery once, they attempted to confirm that if the system had an outage, it would self-recover. It did not. This tripped a very quiet panic at Google because while the company would tick along fine for awhile without access to the master password database, systems would, one by one, fail out if people couldn't get to the passwords that had to be occasionally hand-entered to keep them running. So a cross-continent panic ensued because restarting the database required access to two keycards for NORAD-style simultaneous activation. One was in an executive's wallet who was on vacation, and they had to be flown back to the datacenter to plug it in. The other one was stored in a safe built into the floor of a datacenter, and the combination to that safe was... In the password database. They hired a local safecracker to drill it open, fetched the keycard, double-keyed the initialization machines to reboot the database, and the outside world was none the wiser. (1) I say "ironically," but the actual point of their self-testing is to cause these kinds of disruptions before chance does. They aren't generally supposed to cause user-facing disruption; sometimes they do. Management frowns on disruption in general, but when it's due to disaster recovery testing, they attach to that frown the grain of salt that "Because this failure-mode existed, it would have occurred eventually if it didn't occur today."

hnaccy 5y ago
what if the executive had been pick-pocketed
- shadowgovt 5y ago
  EDIT: I had mis-remembered this part of the story. ;) What was stored in the executive's brain was the combination to a second floor safe in another datacenter that held one of the two necessary activation cards. Whether they were able to pass it to the datacenter over a secure / semi-secure line or flew back to hand-deliver the combination I do not remember. If you mean "Would the pick-pocket have access to valuable Google data," I think the answer is "No, they still don't have the key in the safe on the other continent." If you mean "Would the pick-pocket have created a critical outage at Google that would have required intense amounts of labor to recover from," I don't know because I don't know how many layers of redundancy their recovery protocols had for that outage. It's possible Google came within a hair's breadth of "Thaw out the password database from offline storage, rebuild what can be rebuilt by hand, and inform a smaller subset of the company that some passwords are now just gone and they'll have to recover on their own" territory.

outworlder 5y ago

> I can't fathom how they didn't plan for this Maybe because they were planning for a million other possible things to go wrong, likely with higher probability than this. And busy with each day's pressing matters.

deleted 5y ago
[deleted]
mynameisvlad 5y ago
A config change gone bad? That’s like failure scenarios 101. That should be the second on the list, after “code change gone bad”.
- jfrunyon 5y ago
  Exactly! Obviously they have extremely robust testing and error catching on things like code deploys: how many times do you think they deploy new code a day? And at least personally, their error rate is somewhere below 1%. Clearly something about their networking infrastructure is not as robust.
- pupdogg 5y ago
  Right? Especially on global scale. Something doesn't add up!
  - TheOtherHobbes 5y ago
    Curious/unfortunate timing. The day after a whistleblower docu and with a long list of other legal challenges and issues incoming.
smrtinsert 5y ago
Haha sure. They were too busy implementing php compilers to figure out that "whole DR DNS thing" rotflmao. I'd remove Facebook from my resume.
jfrunyon 5y ago

amalcon 5y ago

Most likely they did plan for this. Then, something happened that the failsafe couldn't handle. E.g. if something overwrites /etc/passwd, having a local user won't help. I'm not saying that specific thing happened here -- it's actually vanishingly unlikely -- but your plan can't cover every contingency.

deleted 5y ago
[deleted]
robalfonso 5y ago
Agreed, it’s also worth mentioning that at the end of every cloud is real physical hardware, and that is decidedly less flexible than cloud, if you locked yourself out of a physical switch or router you have many fewer options.
onecommentman 5y ago
In risk management cultures where consequences from failures are much, much higher, the saying goes that “failsafe systems fail by failing to be failsafe”. Explicit accounting for scenarios where the failsafe fails is a requirement. Great truths of the 1960s to be relearned, I guess.