3 ms·
The S3 checksum feature applies to the objects, so that’s entirely orthogonal to erasure codes. Unless you know something I don’t and SHA256 has commutative pro
by catlifeonmars 3y ago
The S3 checksum feature applies to the objects, so that’s entirely orthogonal to erasure codes. Unless you know something I don’t and SHA256 has commutative properties. You’d still need to compute the object hash independent of any blocks.
Source: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/checki...
- benlivengood 3y agoIt's not entirely orthogonal; RAID5 plus stripe-level CRC (or better) can reliably correct bitrot at any single position in a stripe whereas RAID5 alone can only report an error. My guess is that S3 and other large object stores have the equivalent of stripe-level checksums for this purpose.
- catlifeonmars 3y agoI’m positive something like this is the case. Yet it’s entirely orthogonal to the object hash in the user facing feature, which would need to be computed separately.
- benlivengood 3y agoFor append-only or write-once objects or for BLAKE-3 and other fully parallelizable hashes it's possible to store the intermediate hash function state with each chunk or stripe so that the final bytes of the data, once the hash is finished, yield the user-facing checksum as well.