4 ms·

the catting issue might be more an implementation of bzip program problem than algorithm (it could expect an array of compressed files). that would only be impo

the catting issue might be more an implementation of bzip program problem than algorithm (it could expect an array of compressed files). that would only be impossible if the program cannot reason about the length of data from file header, which again is technically not something about compression algo but rather file format its carried through. that being said, speed is important for compression so for systems like webservers etc its an easy sell ofc. very strong point (and smarter implementation in programs) for gzip

nine_k 8mo ago
Bzip2 is great for files that are compressed once, get decompressed many times, and the size is important. A good example is a software release.
- pocksuppet 8mo ago
  So is xz, or zstd, and the files are smaller. bzip2 disappeared from software releases when xz was widely available. gzip often remains, as the most compatible option, the FAT32 of compression algorithms.
- lucb1e 8mo ago
  Huh? Only if it gets decompressed few times I would say, because it's so extremely slow at it Like a software installation that you do one time. I'd not even want it for updates if I need to update large data files. The only purpose I'd see is the first-time install where users are okay waiting a while, and small code patches that are distributed to a lot of people (Or indeed if size is important, but then again bzip2 only shines if it's text-like. I don't really find this niche knowledge for a few % optimization worth teaching tbh. Better teach general principles so people can find a fitting solution if they ever find themselves under specific constraints like OP)
joecool1029 8mo ago
> the catting issue might be more an implementation of bzip program problem than algorithm (it could expect an array of compressed files). that would only be impossible if the program cannot reason about the length of data from file header, which again is technically not something about compression algo but rather file format its carried through. Long comment to just say: ‘I have no idea about what I’m writing about’ These compression algorithms do not have anything to do with filesystem structure. Anyway the reason you can’t cat together parts of bzip2 but you can with zstd (and gzip) is because zstd does everything in frames and everything in those frames can be decompressed separately (so you can seek and decompress parts). Bzip2 doesn’t do that. So like, another place bzip2 sucks ass is working with large archives because you need to seek the entire archive before you can decompress it and it makes situations without parity data way more likely to cause dataloss of the whole archive. Really, don’t use it unless you have a super specific use case and know the tradeoffs, for the average person it was great when we would spend the time compressing to save the time sending over dialup.
- prerok 8mo ago
  So... it's actually a reasonable objection over bzip2? I mean, you explained why it does not work with bzip2. I think their argument is sound and it makes using bzip2 less useful in certain situations. I was once saved in resolving a problem we had when I figured out that concatening gzipped files just works out of the box. If not, it would have meant a bit more code, lots of additional testing, etc.
  - saidnooneever 7mo ago
    totally agree with the statement though i feel its not an objection over bzip 2 rather than how it was implemented in programs that apply it. but i'm not really 100% since admittedly i did not personally reverse engineer bzip capable programs to see the current state of afairs. I am simply going by descriptions posted in comments and general system knowlesge. how to compress data has little to no relation to how this compression can be implemented in programs. How its implemented, will reflect on how the quality of the algorithm is perceived, becaus e the two are not seperate from a user perspective.
- duskwuff 8mo ago
  > zstd does everything in frames and everything in those frames can be decompressed separately (so you can seek and decompress parts). Bzip2 doesn’t do that. This isn't accurate. 1) Most zstd streams consist of a single frame. The compressor only creates multiple frames if specifically directed to do so. 2) bzip2 blocks, by contrast, are fully independent - by default, the compressor works on 900 kB blocks of input, and each one is stored with no interdependencies between blocks. (However, software support for seeking within the archive is practically nonexistent.)
- saidnooneever 7mo ago
  you misread my comment. i exactly implied the catting issue has relation to FS structure, and hence is not an issue against the bzip algo. sorry if i was unclear.

nine_k 8mo ago

Bzip2 is great for files that are compressed once, get decompressed many times, and the size is important. A good example is a software release.

pocksuppet 8mo ago
So is xz, or zstd, and the files are smaller. bzip2 disappeared from software releases when xz was widely available. gzip often remains, as the most compatible option, the FAT32 of compression algorithms.
lucb1e 8mo ago
Huh? Only if it gets decompressed few times I would say, because it's so extremely slow at it Like a software installation that you do one time. I'd not even want it for updates if I need to update large data files. The only purpose I'd see is the first-time install where users are okay waiting a while, and small code patches that are distributed to a lot of people (Or indeed if size is important, but then again bzip2 only shines if it's text-like. I don't really find this niche knowledge for a few % optimization worth teaching tbh. Better teach general principles so people can find a fitting solution if they ever find themselves under specific constraints like OP)

joecool1029 8mo ago

> the catting issue might be more an implementation of bzip program problem than algorithm (it could expect an array of compressed files). that would only be impossible if the program cannot reason about the length of data from file header, which again is technically not something about compression algo but rather file format its carried through. Long comment to just say: ‘I have no idea about what I’m writing about’ These compression algorithms do not have anything to do with filesystem structure. Anyway the reason you can’t cat together parts of bzip2 but you can with zstd (and gzip) is because zstd does everything in frames and everything in those frames can be decompressed separately (so you can seek and decompress parts). Bzip2 doesn’t do that. So like, another place bzip2 sucks ass is working with large archives because you need to seek the entire archive before you can decompress it and it makes situations without parity data way more likely to cause dataloss of the whole archive. Really, don’t use it unless you have a super specific use case and know the tradeoffs, for the average person it was great when we would spend the time compressing to save the time sending over dialup.

prerok 8mo ago
So... it's actually a reasonable objection over bzip2? I mean, you explained why it does not work with bzip2. I think their argument is sound and it makes using bzip2 less useful in certain situations. I was once saved in resolving a problem we had when I figured out that concatening gzipped files just works out of the box. If not, it would have meant a bit more code, lots of additional testing, etc.
- saidnooneever 7mo ago
  totally agree with the statement though i feel its not an objection over bzip 2 rather than how it was implemented in programs that apply it. but i'm not really 100% since admittedly i did not personally reverse engineer bzip capable programs to see the current state of afairs. I am simply going by descriptions posted in comments and general system knowlesge. how to compress data has little to no relation to how this compression can be implemented in programs. How its implemented, will reflect on how the quality of the algorithm is perceived, becaus e the two are not seperate from a user perspective.
duskwuff 8mo ago
> zstd does everything in frames and everything in those frames can be decompressed separately (so you can seek and decompress parts). Bzip2 doesn’t do that. This isn't accurate. 1) Most zstd streams consist of a single frame. The compressor only creates multiple frames if specifically directed to do so. 2) bzip2 blocks, by contrast, are fully independent - by default, the compressor works on 900 kB blocks of input, and each one is stored with no interdependencies between blocks. (However, software support for seeking within the archive is practically nonexistent.)
saidnooneever 7mo ago
you misread my comment. i exactly implied the catting issue has relation to FS structure, and hence is not an issue against the bzip algo. sorry if i was unclear.