4 ms·
AWS S3 right now supports CRC32, CRC32C, SHA-1, and SHA-256 (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html https://docs.a
by theden 4y ago
AWS S3 right now supports CRC32, CRC32C, SHA-1, and SHA-256 (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/checki...), which is interesting given the announcement to support the four wasn't long ago (https://aws.amazon.com/blogs/aws/new-additional-checksum-algorithms-for-amazon-s3/)—it https://aws.amazon.com/blogs/aws/new-additional-checksum-alg... seems like they went with what's more widely used than actually faster?
- jandrewrogers 4y agoThe performance of these algorithms isn't entirely dependent on software when running at the scale of S3. Dedicated hashing hardware or hardware support will likely have reasonable throughput. You don't need the maximum possible performance, just sufficient to achieve objectives as inexpensively as possible.
- adrian_b 4y agoAll modern Intel/AMD CPUs and 64-bit ARM CPUs have hardware instructions for these 4 algorithms (polynomial multiplication for CRC32, dedicated instructions for the other 3). Therefore these are very fast when using the hardware support, so they are a natural choice versus other algorithms that must be implemented in software, which may be slower even when they seem much simpler.
- foota 4y agoThese are probably chosen based on popularity than anything.