3 ms·
Intuitively, it feels as though grouping is a tradeoff that lowers the probability of a data loss event, at the expense of increasing the amount of data lost in
by jl6 5y ago
Intuitively, it feels as though grouping is a tradeoff that lowers the probability of a data loss event, at the expense of increasing the amount of data lost in a data loss event?
Edit: “proof” by example…
9 disks, replication factor of 3. 9 choose 3 = 84 ways to allocate a file’s replicas across the disks. If 3 disks fail, on average 1/84 of the total data is lost (assuming >>84 files).
Versus… 3 groups of 3 disks. If 3 disks fail, the probability of the second disk failure being in the same group as the first disk failure is 2/8, and then 1/7 that the third failure is also in the same group. (2/8)*(1/7)=1/28.
So a 100% chance of losing 1/84 of the data versus a 1/28 chance of losing 1/3 of the data. Same total expected amount of data lost.
- jerf 5y agoYou begin to see why this is a subtle problem, and why I think it catches so many people. Naive statistics, and perhaps "intuitive" statistics, say it's no different. In reality it's hugely different. It is generally preferable in reality to take the second choice because additional realities intervene to give you time to deal with the problem, allowing the existence of fairly highly-reliable systems, whereas those same additional realities mean that in practice, you will lose data with some frequency if you select the uniformly random result. Sometimes the better systems still catastrophically lose data, that is indeed inevitable as you point out, but what it takes to knock out that system would have result in lost data in the bad uniformly-random system, too, so in practice it's not much of a tradeoff.