Weirdly enough, we see this all the time in large systems. When we built DynamoDB - the core 3-way replication is based on the fact that machine failure rates a

Weirdly enough, we see this all the time in large systems. When we built DynamoDB - the core 3-way replication is based on the fact that machine failure rates are something like 0.005, so we will need to be unlucky (0.005)^2 times. Guess what? it happens alarmingly often enough when you run 100k boxes.