3 ms·

AFAIK (not 100% sure), multithreading support is different - parallel versions split the file in multiple segments, and compress each independently, which multi

AFAIK (not 100% sure), multithreading support is different - parallel versions split the file in multiple segments, and compress each independently, which multithreaded functionalities apply to the same stream (no hard splitting). For this reason, there's for example pzstd, in addition to zstd.

faeyanpiraat 3y ago
tangential question: compressed files look like hashes in terms of if something changes in the beginning, all the other parts are different, right?
- masklinn 3y ago
  Usually but it depends on the compression scheme. There’s usually a “window” of how far back they look, so they can resync after a while, but it’s unlikely and the offsets will likely have changed so you need to handle that. You can force this property by introducing synchronisation points though, gzip has an `—-rsyncable` which makes that a lot more likely, at a small compression cost. Edit: apparently zstd has also had —-rsyncable for the last 5 years.
- nine_k 3y ago
  For compression efficiency, it makes sense to have one large common dictionary. For compression speed, it is easier to have a dictionary per chunk. I still hope they use the common dictionary; if so, any change in the beginning likely affects further parts if it affects the dictionary and thus the way how the later parts are compressed. Same for farther parts affecting the way earlier parts are compressed.
MisterTea 3y ago
Yes, pbzip divides up the file into blocks per core. Though I think some versions (older?) of bzip are unable to handle pbzip archives. I used pbzip2 on an old octo core xeon server with a decent sas raid and was able to compress at well over 200MB/sec, closer to 300MB in some cases.
aidenn0 3y ago
bzip2 is already block based though, so there is no compatibility issue in that specific case (vs pbzip2) though I think pbzip2 supports larger blocks than the original bzip2.

faeyanpiraat 3y ago

tangential question: compressed files look like hashes in terms of if something changes in the beginning, all the other parts are different, right?

masklinn 3y ago
Usually but it depends on the compression scheme. There’s usually a “window” of how far back they look, so they can resync after a while, but it’s unlikely and the offsets will likely have changed so you need to handle that. You can force this property by introducing synchronisation points though, gzip has an `—-rsyncable` which makes that a lot more likely, at a small compression cost. Edit: apparently zstd has also had —-rsyncable for the last 5 years.
nine_k 3y ago
For compression efficiency, it makes sense to have one large common dictionary. For compression speed, it is easier to have a dictionary per chunk. I still hope they use the common dictionary; if so, any change in the beginning likely affects further parts if it affects the dictionary and thus the way how the later parts are compressed. Same for farther parts affecting the way earlier parts are compressed.

MisterTea 3y ago

Yes, pbzip divides up the file into blocks per core. Though I think some versions (older?) of bzip are unable to handle pbzip archives. I used pbzip2 on an old octo core xeon server with a decent sas raid and was able to compress at well over 200MB/sec, closer to 300MB in some cases.

aidenn0 3y ago

bzip2 is already block based though, so there is no compatibility issue in that specific case (vs pbzip2) though I think pbzip2 supports larger blocks than the original bzip2.