4 ms·
Back in Ye Old Dark Ages, I caused a BIG Google outage by running a routine maintenance script that had been run dozens if not hundreds of times before. Turns
by packetslave 5y ago
Back in Ye Old Dark Ages, I caused a BIG Google outage by running a routine maintenance script that had been run dozens if not hundreds of times before.
Turns out the underlying network software had a race condition that would ONLY be hit if the script ran at the exact same time as some automated monitoring tools polled the box.
At FAANG scale, "one in a million" happens a lot more often than you'd think.
- ecnahc515 5y ago> At FAANG scale, "one in a million" happens a lot more often than you'd think. And it happens less than you think too, sometimes, which I think is closer to the original point.