3 ms·
> The AWS S3 Rust SDK was found to be unnecessarily recomputing CRC32C checksums. I'd be interested in hearing what you changed to work around this.
by JoshTriplett 2y ago
> The AWS S3 Rust SDK was found to be unnecessarily recomputing CRC32C checksums.
I'd be interested in hearing what you changed to work around this.
- infiniteregrets 2y agohi, S2 dev here. I found that if you explicitly set the algorithm for the checksum (crc32c) aws SDK would ignore the provided checksum and we were doing both. I also found a related issue https://github.com/awslabs/aws-sdk-rust/issues/1103 https://github.com/awslabs/aws-sdk-rust/issues/1103
- foota 2y agoIn my experience (working on a heavily checksummed system) if you don't carefully think about where you're actually required to take checksums for data integrity, it's very easy to end up with wasteful checksums with every library/system/etc., taking a checksum "just in case", and as a bonus you might miss some important checksums that aren't obvious when transforming data.