3 ms·

I think that per-chunk checksums can be helpful if you are manipulating PNG files. For example imaging that you are "cleaning" a PNG by removing metadata such a

I think that per-chunk checksums can be helpful if you are manipulating PNG files. For example imaging that you are "cleaning" a PNG by removing metadata such as location information and timezone. The checksum can help make sure that you don't mess up your copying of unchanged chunks (or at least you notice your mistake faster). Even if your code is perfect it could help detect a bitflip during processing. Admittedly minor, but why not? I think the biggest mistake is that the checksum doesn't cover the type and length. If it did then I most of your concerns would be resolved. Although it may also make sense to have a full-file checksum in the IEND, but the only thing that could really detect is if whole chunks were perfectly dropped somehow, so not much added value, but again 4 bytes seems worth it.

dalke 4y ago
While I've never done the task you describe, it's similar across many FourCC formats so I think my experience is applicable. I strongly suspect the input checksum won't be checked against the output data. Data ingestion might/should verify the chucksum, which is then thrown away. This is especially true if working in a language with immutable strings, or using a functional-style immutable approach, where it's easier to know the payload doesn't change. The checksum will be recomputed in egress. As an alternative approach, the entire chunk might be stored in a single block, and either filtered or written as a single block, with no need to change anything, so no need to recompute the checksum. In any case, if the developer thought this was appropriate, it's easy to add any sort of checksum or hash fingerprint as part of the chunk reader API, without it being present in the file. > doesn't cover the type and length While it could cover type, length is harder for some use cases. If you have a seekable output file, and don't have the ability to buffer all the data in memory, you might be able to process a segment at a time, write the crc, seek to the beginning of the chunk, then write the size. Oh! I just realized that if the CRC were in the order typecode, data, length (which is different than the presentation order in the PNG data stream) then it would be possible to include the length in the CRC. Though I don't think including the length would improve things as I think the failure modes are identical. Maybe?
- nayuki 4y ago
  If the length comes last, how would you know how much data to read? Remember that chunks like tEXt are variable-length and not self-terminating; it relies on the outer level to signal the end of data.
  - dalke 4y ago
    The length would still come first in the PNG datastream. My idea is that those 4 bytes could have been added to the CRC after processing the chunk tag and data. Here's an example: class BlockWriter: def __init__(self, output, chunk_type): self.output = output self._start = output.tell() output.write(b"\0\0\0\0") output.write(chunk_type) self._crc = zlib.crc32(chunk_type) self._chunk_length = 0 def write(self, data): self.output.write(data) self._crc = zlib.crc32(data, self._crc) self._chunk_length += len(data) def finish(self): chunk_length_bytes = self._chunk_length.to_bytes(4, "big") self._crc = zlib.crc32(chunk_length_bytes, self._crc) # !! NEW !! crc_bytes = self._crc.to_bytes(4, "big") self.output.write(crc_bytes) # Go back and update the length end = self.output.tell() self.output.seek(self._start) self.output.write(chunk_length_bytes) self.output.seek(end) It could be used like: writer = BlockWriter(png_f, b"IDAT") writer.write(b"shrdlu") writer.write(b"etaoin") writer.finish() which produces the correct block structure: '\x00\x00\x00\x0cIDATshrdluetaoin\xed?\xa6\xa4' |--------------| four byte length = 12 |--| four byte character code 'IDAT' |----------| = 12 bytes of payload |------------| = four byte CRC I'll show the CRC is in the order {tag}, {data}, {length}: >>> zlib.crc32(b"IDAT" + b"shrdluetaoin" + b"\x00\x00\x00\x0c" ).to_bytes(4, "big") b'\xed?\xa6\xa4'
nayuki 4y ago
@parent and @grandparent: The chunk CRC-32 does cover the chunk type. > A four-byte CRC (Cyclic Redundancy Code) calculated on the preceding bytes in the chunk, including the chunk type field and chunk data fields, but not including the length field. -- https://www.w3.org/TR/2003/REC-PNG-20031110/#5Chunk-layout https://www.w3.org/TR/2003/REC-PNG-20031110/#5Chunk-layout
- dalke 4y ago
  Thank you for the correction! It has been many years since I looked into this matter, and I seem to have forgotten that detail. Digging through my sent box, to png-mng-misc, I see that knew that back in 2012! I also wrote that most of the tools I checked didn't verify the CRC: > I tried a PNG with an IDAT chunk with an invalid CRC on various software on the my Mac. Unless I messed up my testing, the desktop, email preview, OmniGraffle, and Pixen.app all used the chunk with the invalid CRC. My thread was "I would like some insight about PNG CRC and other experience" in case someone wants to dig it up. I was able to make an PNG with a length field which, if changed, would produce another PNG showing a different result. At least one of the PNGs had an invalid checksum, but was still displayed. EDIT: I found a public copy of the thread at https://png-mng-misc.narkive.com/YgUoUekk/i-would-like-some-insight-about-png-crc-and-other-experience https://png-mng-misc.narkive.com/YgUoUekk/i-would-like-some-... .
  - dalke 4y ago
    For what it's worth, here's an example of two PNGs which differ by 1 bit, in the length. http://dalkescientific.com/xo.png http://dalkescientific.com/xo.png is a PNG for an "O" with two IDAT blocks: 'IHDR' 13 (this is the chunk data size, excluding the 4 bytes of crc) 'xtra' 0 'IDAT' 1012 <-- this one is displayed 'IDAT' 1090 <-- this one is ignored 'IEND' 0 http://dalkescientific.com/xo_bad.png http://dalkescientific.com/xo_bad.png is an invalid PNG for an "X" with one IDAT block and with an invalid checksum: 'IHDR' 13 'xtra' 1024 <-- the extra length contains what was the first IDAT 'IDAT' 1090 <-- this one is displayed 'IEND' 0 Both display in Preview.app, Firefox, and Safari. They differ only in byte offset 35, where one has a chr(0) and the other a chr(4): % python -c 'print(set(enumerate(open("xo.png", "rb").read()))\ .symmetric_difference(\ set(enumerate(open("xo_bad.png", "rb").read()))))' {(35, 0), (35, 4)} With a bit more work I could probably construct PNGs which are both valid, and which differ only by a bit. It would require brute-forcing some chunk data so the CRCs would get a match. I have some ideas on how to make it so there's only one valid (and different) IDAT block as well, but that's more than I can do in an evening.

dalke 4y ago

While I've never done the task you describe, it's similar across many FourCC formats so I think my experience is applicable. I strongly suspect the input checksum won't be checked against the output data. Data ingestion might/should verify the chucksum, which is then thrown away. This is especially true if working in a language with immutable strings, or using a functional-style immutable approach, where it's easier to know the payload doesn't change. The checksum will be recomputed in egress. As an alternative approach, the entire chunk might be stored in a single block, and either filtered or written as a single block, with no need to change anything, so no need to recompute the checksum. In any case, if the developer thought this was appropriate, it's easy to add any sort of checksum or hash fingerprint as part of the chunk reader API, without it being present in the file. > doesn't cover the type and length While it could cover type, length is harder for some use cases. If you have a seekable output file, and don't have the ability to buffer all the data in memory, you might be able to process a segment at a time, write the crc, seek to the beginning of the chunk, then write the size. Oh! I just realized that if the CRC were in the order typecode, data, length (which is different than the presentation order in the PNG data stream) then it would be possible to include the length in the CRC. Though I don't think including the length would improve things as I think the failure modes are identical. Maybe?

nayuki 4y ago
If the length comes last, how would you know how much data to read? Remember that chunks like tEXt are variable-length and not self-terminating; it relies on the outer level to signal the end of data.
- dalke 4y ago
  The length would still come first in the PNG datastream. My idea is that those 4 bytes could have been added to the CRC after processing the chunk tag and data. Here's an example: class BlockWriter: def __init__(self, output, chunk_type): self.output = output self._start = output.tell() output.write(b"\0\0\0\0") output.write(chunk_type) self._crc = zlib.crc32(chunk_type) self._chunk_length = 0 def write(self, data): self.output.write(data) self._crc = zlib.crc32(data, self._crc) self._chunk_length += len(data) def finish(self): chunk_length_bytes = self._chunk_length.to_bytes(4, "big") self._crc = zlib.crc32(chunk_length_bytes, self._crc) # !! NEW !! crc_bytes = self._crc.to_bytes(4, "big") self.output.write(crc_bytes) # Go back and update the length end = self.output.tell() self.output.seek(self._start) self.output.write(chunk_length_bytes) self.output.seek(end) It could be used like: writer = BlockWriter(png_f, b"IDAT") writer.write(b"shrdlu") writer.write(b"etaoin") writer.finish() which produces the correct block structure: '\x00\x00\x00\x0cIDATshrdluetaoin\xed?\xa6\xa4' |--------------| four byte length = 12 |--| four byte character code 'IDAT' |----------| = 12 bytes of payload |------------| = four byte CRC I'll show the CRC is in the order {tag}, {data}, {length}: >>> zlib.crc32(b"IDAT" + b"shrdluetaoin" + b"\x00\x00\x00\x0c" ).to_bytes(4, "big") b'\xed?\xa6\xa4'

nayuki 4y ago

@parent and @grandparent: The chunk CRC-32 does cover the chunk type. > A four-byte CRC (Cyclic Redundancy Code) calculated on the preceding bytes in the chunk, including the chunk type field and chunk data fields, but not including the length field. -- https://www.w3.org/TR/2003/REC-PNG-20031110/#5Chunk-layout https://www.w3.org/TR/2003/REC-PNG-20031110/#5Chunk-layout

dalke 4y ago
Thank you for the correction! It has been many years since I looked into this matter, and I seem to have forgotten that detail. Digging through my sent box, to png-mng-misc, I see that knew that back in 2012! I also wrote that most of the tools I checked didn't verify the CRC: > I tried a PNG with an IDAT chunk with an invalid CRC on various software on the my Mac. Unless I messed up my testing, the desktop, email preview, OmniGraffle, and Pixen.app all used the chunk with the invalid CRC. My thread was "I would like some insight about PNG CRC and other experience" in case someone wants to dig it up. I was able to make an PNG with a length field which, if changed, would produce another PNG showing a different result. At least one of the PNGs had an invalid checksum, but was still displayed. EDIT: I found a public copy of the thread at https://png-mng-misc.narkive.com/YgUoUekk/i-would-like-some-insight-about-png-crc-and-other-experience https://png-mng-misc.narkive.com/YgUoUekk/i-would-like-some-... .
- dalke 4y ago
  For what it's worth, here's an example of two PNGs which differ by 1 bit, in the length. http://dalkescientific.com/xo.png http://dalkescientific.com/xo.png is a PNG for an "O" with two IDAT blocks: 'IHDR' 13 (this is the chunk data size, excluding the 4 bytes of crc) 'xtra' 0 'IDAT' 1012 <-- this one is displayed 'IDAT' 1090 <-- this one is ignored 'IEND' 0 http://dalkescientific.com/xo_bad.png http://dalkescientific.com/xo_bad.png is an invalid PNG for an "X" with one IDAT block and with an invalid checksum: 'IHDR' 13 'xtra' 1024 <-- the extra length contains what was the first IDAT 'IDAT' 1090 <-- this one is displayed 'IEND' 0 Both display in Preview.app, Firefox, and Safari. They differ only in byte offset 35, where one has a chr(0) and the other a chr(4): % python -c 'print(set(enumerate(open("xo.png", "rb").read()))\ .symmetric_difference(\ set(enumerate(open("xo_bad.png", "rb").read()))))' {(35, 0), (35, 4)} With a bit more work I could probably construct PNGs which are both valid, and which differ only by a bit. It would require brute-forcing some chunk data so the CRCs would get a match. I have some ideas on how to make it so there's only one valid (and different) IDAT block as well, but that's more than I can do in an evening.