5 ms·
The seemingly high occurence of rare events is an artifact of how much we're measuring. If you have a thousand independent weather stations, every year you'll
by vikingerik 2y ago
The seemingly high occurence of rare events is an artifact of how much we're measuring. If you have a thousand independent weather stations, every year you'll see a "thousand year" event at one of them. We just didn't notice the other 999 that didn't.
Same deal as how data centers manage equipment failures. If the drives have a million-hour MTBF, but you have a million drives, you're replacing one every hour. Thousand-year events happen all the time when you have a thousand trials.
This one may have been something wilder like 1 in 20k years, and among 1k endpoints you'll see that every 20 years, and this could be comparable to Katrina which was now 20 years ago.
How do you shore up infrastructure: either you spend a ton of money, or you don't do it and deal with the cleanup afterwards. Each additional 9 of reliability costs 10x. At some point the cleanup for one point is less than shoring up all of them.
- unsnap_biceps 2y agoThese are not purely random events. Every weather station doesn't rolls the dice on their own. If I put 1000 weather stations in my back yard, it doesn't suddenly flood every year.
- mbauman 2y ago> The seemingly high occurence of rare events is an artifact of how much we're measuring. If you have a thousand weather stations, every year you'll see a "thousand year" event at one of them. We just didn't notice the other 999 that didn't. That would only be true if all those weather stations are fully independent. They're definitively not. Drives should be independent, but all drives in a datacenter would similarly share their environment. Run the datacenter too hot and your MTBF will be out of spec.