3 ms·

>The CRCs are entirely dead weight today Why? The usual answer is that "checksumming should be part of the FS layer". My usual retort to such an assertion is

>The CRCs are entirely dead weight today Why? The usual answer is that "checksumming should be part of the FS layer". My usual retort to such an assertion is that filesystem checksums won't save you when the data given to the FS layer is already corrupted, due to bit flips in the writer process's memory. I personally have encountered data loss due to faulty RAM (admittedly non-ECC, thanks to Intel) when copying large amounts of data from one machine to another. You need end-to-end integrity checks. Period.

Retr0id 4y ago
I agree with the "usual" answer, or more generally, "the layer above". We shouldn't expect every file format to roll its own error detection. If you truly care about detecting bit-flips in a writer process's memory, that's a very niche use-case - and maybe you should wrap your files in PAR2 (or even just a .zip in store mode!). 99% of in-the-wild PNGs are checksummed or cryptographically signed at a layer above the file format (e.g. as part of a signed software package, or served over SSL). Edit: Furthermore, the PNG image data is already checksummed as part of zlib (with the slightly weaker adler32 checksum), so the second layer of checksumming is mostly redundant.
- remram 4y ago
  > We shouldn't expect every file format to roll its own error detection. On the other hand, why not? If you are dealing with files that are usually 200kB+, putting 4 or 16 bytes towards a checksum is not a big deal and can help in some unusual situations. Even if the decoder ignores it for speed, the cost is very low.
  - Retr0id 4y ago
    The space cost is negligible, but the time cost for the encoder is real. Since most decoders do verify checksums, you can't just skip it. Take fpng[1] as an example, which tries to push the boundaries of PNG encode speed. > The above benchmarks were made before SSE adler32/crc32 functions were added to the encoder. With 24bpp images and MSVC2022 the encoder is now around 15% faster. I can't see the total percentage cost of checksums mentioned anywhere on the page, but we can infer that it's at least 15% of the overall CPU time, on platforms without accelerated checksum implementations. [1] https://github.com/richgel999/fpng https://github.com/richgel999/fpng
    - Nyan 4y ago
      I didn't infer 15% from the way it was written there. But most platforms these days have some form of CRC32 "acceleration". Adler32 is easy to compute so I'm even less concerned there. I spent a bunch of time optimising the code in fpnge (https://github.com/veluca93/fpnge https://github.com/veluca93/fpnge), which is often notably faster than fpng (https://github.com/nigeltao/qoir/blob/5671f584dcf84ddb71e28da6fa60225abe915e43/doc/full_benchmarks.txt#L63-L64 https://github.com/nigeltao/qoir/blob/5671f584dcf84ddb71e28d...), yet checksum time is basically negligible. Having said that, the double-checksum aspect of PNG does feel unnecessary.
    - remram 4y ago
      Does 15% more time to encode matter? How much time is spent encoding files vs decoding? That is probably still an negligible amount of compute, out of the total compute spent on PNGs. Your specific number seem to come from an (old version of) an encoder that has super-optimized encode and not (yet) optimized CRC.
      - Retr0id 4y ago
        If the 15% didn't matter, the optimization wouldn't have been made.
fluoridation 4y ago
CRC can't save you from faulty RAM. It can save you from bitrot in data at rest and from transmission errors. If you have faulty RAM, all bets are off. The data could be corrupted after it's been processed by the CPU (to compute the CRC) and before it's been sent to the storage device. Arguably, the real reason CRC is useless is that most people don't care about the data integrity of their PNGs. Those who do care probably already have a better system of error detection, or maybe even correction.
Dylan16807 4y ago
My retort is "please please give me a filesystem for windows that checksums and isn't highly buggy".

Retr0id 4y ago

I agree with the "usual" answer, or more generally, "the layer above". We shouldn't expect every file format to roll its own error detection. If you truly care about detecting bit-flips in a writer process's memory, that's a very niche use-case - and maybe you should wrap your files in PAR2 (or even just a .zip in store mode!). 99% of in-the-wild PNGs are checksummed or cryptographically signed at a layer above the file format (e.g. as part of a signed software package, or served over SSL). Edit: Furthermore, the PNG image data is already checksummed as part of zlib (with the slightly weaker adler32 checksum), so the second layer of checksumming is mostly redundant.

remram 4y ago
> We shouldn't expect every file format to roll its own error detection. On the other hand, why not? If you are dealing with files that are usually 200kB+, putting 4 or 16 bytes towards a checksum is not a big deal and can help in some unusual situations. Even if the decoder ignores it for speed, the cost is very low.
- Retr0id 4y ago
  The space cost is negligible, but the time cost for the encoder is real. Since most decoders do verify checksums, you can't just skip it. Take fpng[1] as an example, which tries to push the boundaries of PNG encode speed. > The above benchmarks were made before SSE adler32/crc32 functions were added to the encoder. With 24bpp images and MSVC2022 the encoder is now around 15% faster. I can't see the total percentage cost of checksums mentioned anywhere on the page, but we can infer that it's at least 15% of the overall CPU time, on platforms without accelerated checksum implementations. [1] https://github.com/richgel999/fpng https://github.com/richgel999/fpng
  - Nyan 4y ago
    I didn't infer 15% from the way it was written there. But most platforms these days have some form of CRC32 "acceleration". Adler32 is easy to compute so I'm even less concerned there. I spent a bunch of time optimising the code in fpnge (https://github.com/veluca93/fpnge https://github.com/veluca93/fpnge), which is often notably faster than fpng (https://github.com/nigeltao/qoir/blob/5671f584dcf84ddb71e28da6fa60225abe915e43/doc/full_benchmarks.txt#L63-L64 https://github.com/nigeltao/qoir/blob/5671f584dcf84ddb71e28d...), yet checksum time is basically negligible. Having said that, the double-checksum aspect of PNG does feel unnecessary.
  - remram 4y ago
    Does 15% more time to encode matter? How much time is spent encoding files vs decoding? That is probably still an negligible amount of compute, out of the total compute spent on PNGs. Your specific number seem to come from an (old version of) an encoder that has super-optimized encode and not (yet) optimized CRC.
    - Retr0id 4y ago
      If the 15% didn't matter, the optimization wouldn't have been made.

fluoridation 4y ago

CRC can't save you from faulty RAM. It can save you from bitrot in data at rest and from transmission errors. If you have faulty RAM, all bets are off. The data could be corrupted after it's been processed by the CPU (to compute the CRC) and before it's been sent to the storage device. Arguably, the real reason CRC is useless is that most people don't care about the data integrity of their PNGs. Those who do care probably already have a better system of error detection, or maybe even correction.

Dylan16807 4y ago

My retort is "please please give me a filesystem for windows that checksums and isn't highly buggy".