I investigated this bug: Backup system, using a tree data structure where the root was a hash describing a backup, and the leaves were variable-size chunks of d

I investigated this bug: Backup system, using a tree data structure where the root was a hash describing a backup, and the leaves were variable-size chunks of data. Backing up a virtual machine, it would process only the changed areas, and re-build that section of tree. Roughly 1 in a few million backups silently lopped off a branch of the tree, a couple levels up. Customers have thousands of VMs, we have thousands of customers. Silent data corruption, somewhere, every day. Rarely-triggered off-by-one errors in un-reproducable data suck.