3 ms·

It's a tiny issue, but what I like least about the PNG format is the checksum at the end of each block. From what I gather, the checksum is there for two reaso

by dalke 4y ago

It's a tiny issue, but what I like least about the PNG format is the checksum at the end of each block. From what I gather, the checksum is there for two reasons: 1) a check on archival integrity, based on experience with its usefulness in ZIP files, and 2) a way to check for download errors early, before reaching the end-of-file, which was more important in the slow and noisy modem era of the 1990s. However, it's still possible for a change in chunk type to go undetected, for example, if "zTXt" were transformed to "xTXt" - a one bit change. (I think it's also possible to construct a chunk such that if the length changed to just the right value then it could be interpreted as two chunks (with a smaller length) or be merged with later chunks. This requires getting the CRCs to align just right, and even harder to have just a single bit change.) My belief is that removing the per-block CRC32 and putting the checksum in the IEND at the very end of the PNG data stream, and using a stronger checksum - even MD5 - would be more effective at archival integrity. This of course can't happen now. Still, I regard it as a small bit of 1990s cruft. When I developed my own format, for non-image data, I started with PNG as a guideline, then found that dealing with the checksum, even just to always generate a valid value, was a nuisance, with seemingly no good reason to justify its overhead. I decided to drop the checksum, with a hand-waving argument that people should use other tools to detect and even repair file corruption, depending on their specific requirements. This more generic and widespread family of formats is called "FourCC" for "four-character code" (https://en.wikipedia.org/wiki/FourCC https://en.wikipedia.org/wiki/FourCC).

edflsafoiewq 4y ago
The check value is doubly stupid because there are two of them: zlib compression adds the Adler32 of the uncompressed data, and then the IDAT adds a CRC32 of the compressed data.
- layer8 4y ago
  Not all chunk types use compression. The CRC is a generic mechanism that is agnostic of the chunk type.
kevincox 4y ago
I think that per-chunk checksums can be helpful if you are manipulating PNG files. For example imaging that you are "cleaning" a PNG by removing metadata such as location information and timezone. The checksum can help make sure that you don't mess up your copying of unchanged chunks (or at least you notice your mistake faster). Even if your code is perfect it could help detect a bitflip during processing. Admittedly minor, but why not? I think the biggest mistake is that the checksum doesn't cover the type and length. If it did then I most of your concerns would be resolved. Although it may also make sense to have a full-file checksum in the IEND, but the only thing that could really detect is if whole chunks were perfectly dropped somehow, so not much added value, but again 4 bytes seems worth it.
- dalke 4y ago
  While I've never done the task you describe, it's similar across many FourCC formats so I think my experience is applicable. I strongly suspect the input checksum won't be checked against the output data. Data ingestion might/should verify the chucksum, which is then thrown away. This is especially true if working in a language with immutable strings, or using a functional-style immutable approach, where it's easier to know the payload doesn't change. The checksum will be recomputed in egress. As an alternative approach, the entire chunk might be stored in a single block, and either filtered or written as a single block, with no need to change anything, so no need to recompute the checksum. In any case, if the developer thought this was appropriate, it's easy to add any sort of checksum or hash fingerprint as part of the chunk reader API, without it being present in the file. > doesn't cover the type and length While it could cover type, length is harder for some use cases. If you have a seekable output file, and don't have the ability to buffer all the data in memory, you might be able to process a segment at a time, write the crc, seek to the beginning of the chunk, then write the size. Oh! I just realized that if the CRC were in the order typecode, data, length (which is different than the presentation order in the PNG data stream) then it would be possible to include the length in the CRC. Though I don't think including the length would improve things as I think the failure modes are identical. Maybe?
  - nayuki 4y ago

by dalke 4y ago

edflsafoiewq 4y ago

The check value is doubly stupid because there are two of them: zlib compression adds the Adler32 of the uncompressed data, and then the IDAT adds a CRC32 of the compressed data.

layer8 4y ago
Not all chunk types use compression. The CRC is a generic mechanism that is agnostic of the chunk type.

kevincox 4y ago

I think that per-chunk checksums can be helpful if you are manipulating PNG files. For example imaging that you are "cleaning" a PNG by removing metadata such as location information and timezone. The checksum can help make sure that you don't mess up your copying of unchanged chunks (or at least you notice your mistake faster). Even if your code is perfect it could help detect a bitflip during processing. Admittedly minor, but why not? I think the biggest mistake is that the checksum doesn't cover the type and length. If it did then I most of your concerns would be resolved. Although it may also make sense to have a full-file checksum in the IEND, but the only thing that could really detect is if whole chunks were perfectly dropped somehow, so not much added value, but again 4 bytes seems worth it.

dalke 4y ago
While I've never done the task you describe, it's similar across many FourCC formats so I think my experience is applicable. I strongly suspect the input checksum won't be checked against the output data. Data ingestion might/should verify the chucksum, which is then thrown away. This is especially true if working in a language with immutable strings, or using a functional-style immutable approach, where it's easier to know the payload doesn't change. The checksum will be recomputed in egress. As an alternative approach, the entire chunk might be stored in a single block, and either filtered or written as a single block, with no need to change anything, so no need to recompute the checksum. In any case, if the developer thought this was appropriate, it's easy to add any sort of checksum or hash fingerprint as part of the chunk reader API, without it being present in the file. > doesn't cover the type and length While it could cover type, length is harder for some use cases. If you have a seekable output file, and don't have the ability to buffer all the data in memory, you might be able to process a segment at a time, write the crc, seek to the beginning of the chunk, then write the size. Oh! I just realized that if the CRC were in the order typecode, data, length (which is different than the presentation order in the PNG data stream) then it would be possible to include the length in the CRC. Though I don't think including the length would improve things as I think the failure modes are identical. Maybe?
- nayuki 4y ago