2 ms·
tangential question: compressed files look like hashes in terms of if something changes in the beginning, all the other parts are different, right?
by faeyanpiraat 3y ago
tangential question: compressed files look like hashes in terms of if something changes in the beginning, all the other parts are different, right?
- masklinn 3y agoUsually but it depends on the compression scheme. There’s usually a “window” of how far back they look, so they can resync after a while, but it’s unlikely and the offsets will likely have changed so you need to handle that. You can force this property by introducing synchronisation points though, gzip has an `—-rsyncable` which makes that a lot more likely, at a small compression cost. Edit: apparently zstd has also had —-rsyncable for the last 5 years.
- nine_k 3y agoFor compression efficiency, it makes sense to have one large common dictionary. For compression speed, it is easier to have a dictionary per chunk. I still hope they use the common dictionary; if so, any change in the beginning likely affects further parts if it affects the dictionary and thus the way how the later parts are compressed. Same for farther parts affecting the way earlier parts are compressed.