3 ms·

While I've never done the task you describe, it's similar across many FourCC formats so I think my experience is applicable. I strongly suspect the input check

by dalke 4y ago

While I've never done the task you describe, it's similar across many FourCC formats so I think my experience is applicable. I strongly suspect the input checksum won't be checked against the output data. Data ingestion might/should verify the chucksum, which is then thrown away. This is especially true if working in a language with immutable strings, or using a functional-style immutable approach, where it's easier to know the payload doesn't change. The checksum will be recomputed in egress. As an alternative approach, the entire chunk might be stored in a single block, and either filtered or written as a single block, with no need to change anything, so no need to recompute the checksum. In any case, if the developer thought this was appropriate, it's easy to add any sort of checksum or hash fingerprint as part of the chunk reader API, without it being present in the file. > doesn't cover the type and length While it could cover type, length is harder for some use cases. If you have a seekable output file, and don't have the ability to buffer all the data in memory, you might be able to process a segment at a time, write the crc, seek to the beginning of the chunk, then write the size. Oh! I just realized that if the CRC were in the order typecode, data, length (which is different than the presentation order in the PNG data stream) then it would be possible to include the length in the CRC. Though I don't think including the length would improve things as I think the failure modes are identical. Maybe?

nayuki 4y ago
If the length comes last, how would you know how much data to read? Remember that chunks like tEXt are variable-length and not self-terminating; it relies on the outer level to signal the end of data.
- dalke 4y ago
  The length would still come first in the PNG datastream. My idea is that those 4 bytes could have been added to the CRC after processing the chunk tag and data. Here's an example: class BlockWriter: def __init__(self, output, chunk_type): self.output = output self._start = output.tell() output.write(b"\0\0\0\0") output.write(chunk_type) self._crc = zlib.crc32(chunk_type) self._chunk_length = 0 def write(self, data): self.output.write(data) self._crc = zlib.crc32(data, self._crc) self._chunk_length += len(data) def finish(self): chunk_length_bytes = self._chunk_length.to_bytes(4, "big") self._crc = zlib.crc32(chunk_length_bytes, self._crc) # !! NEW !! crc_bytes = self._crc.to_bytes(4, "big") self.output.write(crc_bytes) # Go back and update the length end = self.output.tell() self.output.seek(self._start) self.output.write(chunk_length_bytes) self.output.seek(end) It could be used like: writer = BlockWriter(png_f, b"IDAT") writer.write(b"shrdlu") writer.write(b"etaoin") writer.finish() which produces the correct block structure: '\x00\x00\x00\x0cIDATshrdluetaoin\xed?\xa6\xa4' |--------------| four byte length = 12 |--| four byte character code 'IDAT' |----------| = 12 bytes of payload |------------| = four byte CRC I'll show the CRC is in the order {tag}, {data}, {length}: >>> zlib.crc32(b"IDAT" + b"shrdluetaoin" + b"\x00\x00\x00\x0c" ).to_bytes(4, "big") b'\xed?\xa6\xa4'

by dalke 4y ago

nayuki 4y ago

If the length comes last, how would you know how much data to read? Remember that chunks like tEXt are variable-length and not self-terminating; it relies on the outer level to signal the end of data.

dalke 4y ago
The length would still come first in the PNG datastream. My idea is that those 4 bytes could have been added to the CRC after processing the chunk tag and data. Here's an example: class BlockWriter: def __init__(self, output, chunk_type): self.output = output self._start = output.tell() output.write(b"\0\0\0\0") output.write(chunk_type) self._crc = zlib.crc32(chunk_type) self._chunk_length = 0 def write(self, data): self.output.write(data) self._crc = zlib.crc32(data, self._crc) self._chunk_length += len(data) def finish(self): chunk_length_bytes = self._chunk_length.to_bytes(4, "big") self._crc = zlib.crc32(chunk_length_bytes, self._crc) # !! NEW !! crc_bytes = self._crc.to_bytes(4, "big") self.output.write(crc_bytes) # Go back and update the length end = self.output.tell() self.output.seek(self._start) self.output.write(chunk_length_bytes) self.output.seek(end) It could be used like: writer = BlockWriter(png_f, b"IDAT") writer.write(b"shrdlu") writer.write(b"etaoin") writer.finish() which produces the correct block structure: '\x00\x00\x00\x0cIDATshrdluetaoin\xed?\xa6\xa4' |--------------| four byte length = 12 |--| four byte character code 'IDAT' |----------| = 12 bytes of payload |------------| = four byte CRC I'll show the CRC is in the order {tag}, {data}, {length}: >>> zlib.crc32(b"IDAT" + b"shrdluetaoin" + b"\x00\x00\x00\x0c" ).to_bytes(4, "big") b'\xed?\xa6\xa4'