4 ms·

Yeah, interesting tidbit. It might sound insane today that one engineer's desktop computer could cause such an outage. But that was probably more commonplace 20

Yeah, interesting tidbit. It might sound insane today that one engineer's desktop computer could cause such an outage. But that was probably more commonplace 20 years ago and even today in smaller orgs.

shadowgovt 3y ago
There was a famous incident at one point where code search internally went down. It turned out that while they had deployed the tool internally, one piece of the indexing process was still running as a cron job on the original developer's desktop machine. He went on vacation, his credentials aged out, and of crawler stopped refreshing the index. But my favorite incident will forever be the time they had to drill out a safe because they were disaster-testing the password vault system and discovered that the key needed to restore the password vault system was stored in s aafe, the combination for which had been moved into the password vault system. Only with advanced, modern technology can you lock the keys to the safe in the safe itself with so many steps!
- tomcam 3y ago
  > But my favorite incident will forever be the time they had to drill out a safe because they were disaster-testing the password vault system Great story to be sure—-but I’m going to call it a success. They did the end-to-end testing and caught it then instead of real life.
  - shadowgovt 3y ago
    Well, mostly-kinda-sorta. ;) It's the internal password vault, and there's only one of them, so it's more like "they broke it on purpose and then had to fix it before the company went off the rails." Among the things kept in that vault are credentials that if they age out or aren't regularly refreshed, key internal infrastructure starts grinding to a halt. But still, "it broke while engineers were staring at it and trying to break it" is a better scenario than "it broke surprisingly while engineers were trying to do something else."

shadowgovt 3y ago

There was a famous incident at one point where code search internally went down. It turned out that while they had deployed the tool internally, one piece of the indexing process was still running as a cron job on the original developer's desktop machine. He went on vacation, his credentials aged out, and of crawler stopped refreshing the index. But my favorite incident will forever be the time they had to drill out a safe because they were disaster-testing the password vault system and discovered that the key needed to restore the password vault system was stored in s aafe, the combination for which had been moved into the password vault system. Only with advanced, modern technology can you lock the keys to the safe in the safe itself with so many steps!

tomcam 3y ago
> But my favorite incident will forever be the time they had to drill out a safe because they were disaster-testing the password vault system Great story to be sure—-but I’m going to call it a success. They did the end-to-end testing and caught it then instead of real life.
- shadowgovt 3y ago
  Well, mostly-kinda-sorta. ;) It's the internal password vault, and there's only one of them, so it's more like "they broke it on purpose and then had to fix it before the company went off the rails." Among the things kept in that vault are credentials that if they age out or aren't regularly refreshed, key internal infrastructure starts grinding to a halt. But still, "it broke while engineers were staring at it and trying to break it" is a better scenario than "it broke surprisingly while engineers were trying to do something else."