3 ms·

"We once narrowly missed a major outage because the engineer who submitted the would-be-triggering change unplugged their desktop computer before the change cou

"We once narrowly missed a major outage because the engineer who submitted the would-be-triggering change unplugged their desktop computer before the change could propagate". Sorry, what?

jrms 3y ago
I thought the same.
jedberg 3y ago
The change was being orchestrated from their desktop, and they noticed thing were going sideways, so they unplugged their desktop to stop the deployment. Aka "pressed the big red button".
francisofascii 3y ago
Yeah, interesting tidbit. It might sound insane today that one engineer's desktop computer could cause such an outage. But that was probably more commonplace 20 years ago and even today in smaller orgs.
- shadowgovt 3y ago
  There was a famous incident at one point where code search internally went down. It turned out that while they had deployed the tool internally, one piece of the indexing process was still running as a cron job on the original developer's desktop machine. He went on vacation, his credentials aged out, and of crawler stopped refreshing the index. But my favorite incident will forever be the time they had to drill out a safe because they were disaster-testing the password vault system and discovered that the key needed to restore the password vault system was stored in s aafe, the combination for which had been moved into the password vault system. Only with advanced, modern technology can you lock the keys to the safe in the safe itself with so many steps!
  - tomcam 3y ago
    > But my favorite incident will forever be the time they had to drill out a safe because they were disaster-testing the password vault system Great story to be sure—-but I’m going to call it a success. They did the end-to-end testing and caught it then instead of real life.
    - shadowgovt 3y ago
      Well, mostly-kinda-sorta. ;) It's the internal password vault, and there's only one of them, so it's more like "they broke it on purpose and then had to fix it before the company went off the rails." Among the things kept in that vault are credentials that if they age out or aren't regularly refreshed, key internal infrastructure starts grinding to a halt. But still, "it broke while engineers were staring at it and trying to break it" is a better scenario than "it broke surprisingly while engineers were trying to do something else."
dilyevsky 3y ago
At one point I had to run a script on a substantial portion of their server fleet (like hundreds of thousands machines) and I remember I ran it with a pssh-style utility from desktop (was 10y ago so dunno if they still use this). It was surprisingly quick to do it this way. Could’ve been something like that
codemac 3y ago
It always cracks me up how Google is simultaneously the most web-based company in the world, but their internal political landscape was (Infra, Search, Ads) > everything else. This leads to infra swe writing stupid CLIs all day, rather than having any literal buttons. Things were changing a lot by the time I left though. I do think Google should be more open about their internal outages. This one in particular was very famous internally.
- dilyevsky 3y ago
  We also avoided some outages by running one-off scripts fleet-wide so it cuts both ways
- throwawaaarrgh 3y ago
  Every large enterprise is internally a tire fire.
jeffbee 3y ago
It's a logical consequence of the "zero trust" network. If an engineer's workstation can make RPCs to production systems, and that engineer is properly entitled to assume some privileged role, then there's no difference between running the automation in prod and running it on your workstation. Even at huge scales, shell tools plus RPC client CLIs can contact every machine in the world pretty promptly.
- kevan 3y ago
  There's still differences. If you're running it in prod then the functionality has at least gone through code review and you have higher confidence what's running is what you think it is. If you run things from personal boxes there's always the risk of them not having the latest code, having made a local change and not checking it in, or the worst case of a bad actor doing whatever they want with the privileged role. But if code review isn't required or engineers have unrestricted SSH access to prod hosts then it's pretty much equivalent.
NiloCK 3y ago
To think - if it'd been a laptop they would have had to smash it with a hammer.

"We once narrowly missed a major outage because the engineer who submitted the would-be-triggering change unplugged their desktop computer before the change could propagate". Sorry, what?

jrms 3y ago

I thought the same.

jedberg 3y ago

The change was being orchestrated from their desktop, and they noticed thing were going sideways, so they unplugged their desktop to stop the deployment. Aka "pressed the big red button".

francisofascii 3y ago

Yeah, interesting tidbit. It might sound insane today that one engineer's desktop computer could cause such an outage. But that was probably more commonplace 20 years ago and even today in smaller orgs.

shadowgovt 3y ago
There was a famous incident at one point where code search internally went down. It turned out that while they had deployed the tool internally, one piece of the indexing process was still running as a cron job on the original developer's desktop machine. He went on vacation, his credentials aged out, and of crawler stopped refreshing the index. But my favorite incident will forever be the time they had to drill out a safe because they were disaster-testing the password vault system and discovered that the key needed to restore the password vault system was stored in s aafe, the combination for which had been moved into the password vault system. Only with advanced, modern technology can you lock the keys to the safe in the safe itself with so many steps!
- tomcam 3y ago
  > But my favorite incident will forever be the time they had to drill out a safe because they were disaster-testing the password vault system Great story to be sure—-but I’m going to call it a success. They did the end-to-end testing and caught it then instead of real life.
  - shadowgovt 3y ago
    Well, mostly-kinda-sorta. ;) It's the internal password vault, and there's only one of them, so it's more like "they broke it on purpose and then had to fix it before the company went off the rails." Among the things kept in that vault are credentials that if they age out or aren't regularly refreshed, key internal infrastructure starts grinding to a halt. But still, "it broke while engineers were staring at it and trying to break it" is a better scenario than "it broke surprisingly while engineers were trying to do something else."

dilyevsky 3y ago

At one point I had to run a script on a substantial portion of their server fleet (like hundreds of thousands machines) and I remember I ran it with a pssh-style utility from desktop (was 10y ago so dunno if they still use this). It was surprisingly quick to do it this way. Could’ve been something like that

codemac 3y ago

It always cracks me up how Google is simultaneously the most web-based company in the world, but their internal political landscape was (Infra, Search, Ads) > everything else. This leads to infra swe writing stupid CLIs all day, rather than having any literal buttons. Things were changing a lot by the time I left though. I do think Google should be more open about their internal outages. This one in particular was very famous internally.

dilyevsky 3y ago
We also avoided some outages by running one-off scripts fleet-wide so it cuts both ways
throwawaaarrgh 3y ago
Every large enterprise is internally a tire fire.

jeffbee 3y ago

It's a logical consequence of the "zero trust" network. If an engineer's workstation can make RPCs to production systems, and that engineer is properly entitled to assume some privileged role, then there's no difference between running the automation in prod and running it on your workstation. Even at huge scales, shell tools plus RPC client CLIs can contact every machine in the world pretty promptly.

kevan 3y ago
There's still differences. If you're running it in prod then the functionality has at least gone through code review and you have higher confidence what's running is what you think it is. If you run things from personal boxes there's always the risk of them not having the latest code, having made a local change and not checking it in, or the worst case of a bad actor doing whatever they want with the privileged role. But if code review isn't required or engineers have unrestricted SSH access to prod hosts then it's pretty much equivalent.

NiloCK 3y ago

To think - if it'd been a laptop they would have had to smash it with a hammer.