> you need to be able to move data faster than bad software gets released. And releasing software at a rate of one machine per second still means a release take

> you need to be able to move data faster than bad software gets released. And releasing software at a rate of one machine per second still means a release takes 3 hours! Since others have already demonstrated why the remainder of your comment is overly simplistic, I'll tackle this bit. Generally, software releases are not rolled out at a constant rate to all machines. A typical thing to do is to release it to staging, then to a "canary" subset of machines (e.g. to 1% or 5% of the machines).. Once all seems well there (e.g. metrics are clean and the canaries have handled X writes, reads, and simulated drive failures), it can be rolled out to a larger subset, and eventually to all machines. In that way, the release can take whatever total amount of time is desired while still catching any such bugs fairly reliably. Ideally, at backlblaze they could ensure that their canary instances are "data-redundancy aware" such that even if the 5% they roll to for the canary test all explode, data is still safe. Regardless, any talk of "recovering data faster than software releases" is completely silly and totally misses the reality of how releases are done, how recovery is done, and what sort of bugs might happen. The math based on faulty assumptions about rate is also pointless.