4 ms·

> the catting issue might be more an implementation of bzip program problem than algorithm (it could expect an array of compressed files). that would only be im

> the catting issue might be more an implementation of bzip program problem than algorithm (it could expect an array of compressed files). that would only be impossible if the program cannot reason about the length of data from file header, which again is technically not something about compression algo but rather file format its carried through. Long comment to just say: ‘I have no idea about what I’m writing about’ These compression algorithms do not have anything to do with filesystem structure. Anyway the reason you can’t cat together parts of bzip2 but you can with zstd (and gzip) is because zstd does everything in frames and everything in those frames can be decompressed separately (so you can seek and decompress parts). Bzip2 doesn’t do that. So like, another place bzip2 sucks ass is working with large archives because you need to seek the entire archive before you can decompress it and it makes situations without parity data way more likely to cause dataloss of the whole archive. Really, don’t use it unless you have a super specific use case and know the tradeoffs, for the average person it was great when we would spend the time compressing to save the time sending over dialup.

prerok 7mo ago
So... it's actually a reasonable objection over bzip2? I mean, you explained why it does not work with bzip2. I think their argument is sound and it makes using bzip2 less useful in certain situations. I was once saved in resolving a problem we had when I figured out that concatening gzipped files just works out of the box. If not, it would have meant a bit more code, lots of additional testing, etc.
- saidnooneever 7mo ago
  totally agree with the statement though i feel its not an objection over bzip 2 rather than how it was implemented in programs that apply it. but i'm not really 100% since admittedly i did not personally reverse engineer bzip capable programs to see the current state of afairs. I am simply going by descriptions posted in comments and general system knowlesge. how to compress data has little to no relation to how this compression can be implemented in programs. How its implemented, will reflect on how the quality of the algorithm is perceived, becaus e the two are not seperate from a user perspective.
duskwuff 7mo ago
> zstd does everything in frames and everything in those frames can be decompressed separately (so you can seek and decompress parts). Bzip2 doesn’t do that. This isn't accurate. 1) Most zstd streams consist of a single frame. The compressor only creates multiple frames if specifically directed to do so. 2) bzip2 blocks, by contrast, are fully independent - by default, the compressor works on 900 kB blocks of input, and each one is stored with no interdependencies between blocks. (However, software support for seeking within the archive is practically nonexistent.)
saidnooneever 7mo ago
you misread my comment. i exactly implied the catting issue has relation to FS structure, and hence is not an issue against the bzip algo. sorry if i was unclear.

prerok 7mo ago

So... it's actually a reasonable objection over bzip2? I mean, you explained why it does not work with bzip2. I think their argument is sound and it makes using bzip2 less useful in certain situations. I was once saved in resolving a problem we had when I figured out that concatening gzipped files just works out of the box. If not, it would have meant a bit more code, lots of additional testing, etc.

saidnooneever 7mo ago
totally agree with the statement though i feel its not an objection over bzip 2 rather than how it was implemented in programs that apply it. but i'm not really 100% since admittedly i did not personally reverse engineer bzip capable programs to see the current state of afairs. I am simply going by descriptions posted in comments and general system knowlesge. how to compress data has little to no relation to how this compression can be implemented in programs. How its implemented, will reflect on how the quality of the algorithm is perceived, becaus e the two are not seperate from a user perspective.

duskwuff 7mo ago

> zstd does everything in frames and everything in those frames can be decompressed separately (so you can seek and decompress parts). Bzip2 doesn’t do that. This isn't accurate. 1) Most zstd streams consist of a single frame. The compressor only creates multiple frames if specifically directed to do so. 2) bzip2 blocks, by contrast, are fully independent - by default, the compressor works on 900 kB blocks of input, and each one is stored with no interdependencies between blocks. (However, software support for seeking within the archive is practically nonexistent.)

saidnooneever 7mo ago

you misread my comment. i exactly implied the catting issue has relation to FS structure, and hence is not an issue against the bzip algo. sorry if i was unclear.