6 ms·

An ode to bzip

170 points by signa11 8mo ago

elophanto_agent 8mo ago
[dead]
- kergonath 8mo ago
  Bzip2 is slow. That’s the main issue. Gzip is good enough and much faster. Also, the fact that you cannot get a valid bzip2 file by cat-ing 2 compressed files is not a deal breaker, but it is annoying.
  - saidnooneever 8mo ago
    the catting issue might be more an implementation of bzip program problem than algorithm (it could expect an array of compressed files). that would only be impossible if the program cannot reason about the length of data from file header, which again is technically not something about compression algo but rather file format its carried through. that being said, speed is important for compression so for systems like webservers etc its an easy sell ofc. very strong point (and smarter implementation in programs) for gzip
    - nine_k 8mo ago
      Bzip2 is great for files that are compressed once, get decompressed many times, and the size is important. A good example is a software release.
      - pocksuppet 8mo ago
        So is xz, or zstd, and the files are smaller. bzip2 disappeared from software releases when xz was widely available. gzip often remains, as the most compatible option, the FAT32 of compression algorithms.
      - lucb1e 8mo ago
        Huh? Only if it gets decompressed few times I would say, because it's so extremely slow at it Like a software installation that you do one time. I'd not even want it for updates if I need to update large data files. The only purpose I'd see is the first-time install where users are okay waiting a while, and small code patches that are distributed to a lot of people (Or indeed if size is important, but then again bzip2 only shines if it's text-like. I don't really find this niche knowledge for a few % optimization worth teaching tbh. Better teach general principles so people can find a fitting solution if they ever find themselves under specific constraints like OP)
    - joecool1029 8mo ago
      > the catting issue might be more an implementation of bzip program problem than algorithm (it could expect an array of compressed files). that would only be impossible if the program cannot reason about the length of data from file header, which again is technically not something about compression algo but rather file format its carried through. Long comment to just say: ‘I have no idea about what I’m writing about’ These compression algorithms do not have anything to do with filesystem structure. Anyway the reason you can’t cat together parts of bzip2 but you can with zstd (and gzip) is because zstd does everything in frames and everything in those frames can be decompressed separately (so you can seek and decompress parts). Bzip2 doesn’t do that. So like, another place bzip2 sucks ass is working with large archives because you need to seek the entire archive before you can decompress it and it makes situations without parity data way more likely to cause dataloss of the whole archive. Really, don’t use it unless you have a super specific use case and know the tradeoffs, for the average person it was great when we would spend the time compressing to save the time sending over dialup.
      - prerok 8mo ago
        So... it's actually a reasonable objection over bzip2? I mean, you explained why it does not work with bzip2. I think their argument is sound and it makes using bzip2 less useful in certain situations. I was once saved in resolving a problem we had when I figured out that concatening gzipped files just works out of the box. If not, it would have meant a bit more code, lots of additional testing, etc.
        saidnooneever 7mo ago
        totally agree with the statement though i feel its not an objection over bzip 2 rather than how it was implemented in programs that apply it. but i'm not really 100% since admittedly i did not personally reverse engineer bzip capable programs to see the current state of afairs. I am simply going by descriptions posted in comments and general system knowlesge. how to compress data has little to no relation to how this compression can be implemented in programs. How its implemented, will reflect on how the quality of the algorithm is perceived, becaus e the two are not seperate from a user perspective.
  - nine_k 8mo ago
    Gzip is woefully old. Its only redeeming value is that it's already built into some old tools. Otherwise, use zstd, which is better and faster, both at compression and decompression. There's no reason to use gzip in anything new, except for backwards compatibility with something old.
    - kergonath 8mo ago
      > Otherwise, use zstd, which is better and faster Yes, I do. Zstd is my preferred solution nowadays. But gzip is not going anywhere as a fallback because there is a surprisingly high number of computers without a working libzstd.
    - duskwuff 8mo ago
      One other redeeming quality that gzip/deflate does have is that its low memory requirements (~32 KB per stream). If you're running on an embedded device, or if you're serving a ton of compressed streams at the same time, this can be a meaningful benefit.
    - adrian_b 8mo ago
      The claim that zstd is "better and faster", without additional qualifications, is false and misleading. Indeed for many simple test cases zstd is both better and faster. Despite that, it is possible to find input files for which zstd is either worse or slower than gzip, for any zstd options. I have tested zstd a lot, but eventually I have chosen to use lrzip+gzip for certain very large files (tens of GB file size) with which I am working, because zstd was always either worse or slower. (For those files, at the same compression ratio and on a 6-year old PC, lrzip+gzip has a compression speed always greater than 100 MB/s, while zstd only of 30 to 40 MB/s.) There are also other applications, where I do use zstd, but no compressing program is better than all the others in ALL applications.
  - stefan_ 8mo ago
    bzip and gzip are both horrible, terribly slow. Wherever I see "gz" or "bz" I immediately rip that nonsense out for zstd. There is such a thing as a right choice, and zstd is it every time.
    - laurencerowe 8mo ago
      lz4 can still be the right choice when decompression speed matters. It's almost twice as fast at decompression with similar compression ratios to zstd's fast setting. https://github.com/facebook/zstd?tab=readme-ov-file#benchmarks https://github.com/facebook/zstd?tab=readme-ov-file#benchmar...
    - anthk 8mo ago
      pigz it's damn fast on compressing. Also, a Vax with NetBSD can run gzip. So here is it. Go try these new fancy formats on a Vax, I dare you. And, yes, I prefer LZMA over the obsolete Bzip2 any day, but GZIP it's like the ZIP of free formats modulo packaging, which it's the job of TAR.
    - kergonath 8mo ago
      > Wherever I see "gz" or "bz" That should not happen too often, considering that IIRC bzip lasted only a couple of months before being replaced by bzip2.
  - sedatk 8mo ago
    > the fact that you cannot get a valid bzip2 file by cat-ing 2 compressed files TIL. Now that's why gzip has a file header! But, tar.gz compresses even better, that's probably why it hasn't caught on.
    - pocksuppet 8mo ago
      tar packs multiple files into one. If you concatenate two gzipped files and unzip them, you just get a concatenated file.
      - sedatk 8mo ago
        Ah okay, I thought gzip would support decompressing multiple files that way.
        kergonath 8mo ago
        How it works is, if you have two files foo.gz and bar.gz, and cat foo.gz bar.gz > foobar.gz, then foobar.gz is a valid gzip file and uncompresses to a single file with the contents of foo and bar. It’s handy because it is very easy to just append stuff at the end of a compressed file without having to uncompress-append-recompress. It is a bit niche but I have a couple of use cases where it makes everything simpler.
        bmacho 8mo ago
        tar supports that types of concatenation, so you can concatenate tar.gz files, and unpack them all into separate files
        sedatk 8mo ago
        I know, but I've been always confused why a gzip file would have a filename field in its header if it's supposed to contain only one file. Obviously it's good to keep a backup of original filename somewhere, but it's confusing nonetheless.
  - duskwuff 8mo ago
    bzip2 is particularly slow because the transform it depends on (BWT2) is "intrinsically slow" - it depends on cache-unfriendly operations with long dependency chains, preventing the CPU from extracting any parallelism: https://cbloomrants.blogspot.com/2021/03/faster-inverse-bwt.html https://cbloomrants.blogspot.com/2021/03/faster-inverse-bwt....
- singpolyma3 8mo ago
  Neither has been good enough for years.
joecool1029 8mo ago
Just use zstd unless you absolutely need to save a tiny bit more space. bzip2 and xz are extremely slow to compress.
- silisili 8mo ago
  I'd argue it's more workload dependent, and everything is a tradeoff. In my own testing of compressing internal generic json blobs, I found brotli a clear winner when comparing space and time. If I want higher compatibility and fast speeds, I'd probably just reach for gzip. zstd is good for many use cases, too, perhaps even most...but I think just telling everyone to always use it isn't necessarily the best advice.
  - joecool1029 8mo ago
    > If I want higher compatibility and fast speeds, I'd probably just reach for gzip. It’s slower and compresses less than zstd. gzip should only be reached for as a compatibility option, that’s the only place it wins, it’s everywhere. EDIT: If you must use it, use the modern implementation, https://www.zlib.net/pigz/ https://www.zlib.net/pigz/
    - adrian_b 8mo ago
      Any claims about compressing programs are extremely data-dependent so any general claims will be false for certain test cases. I do a lot of data compression and decompression, and I would have liked a lot to find a magic algorithm that works better than all others, to simplify my work. After extensive tests I have found no such algorithm. Depending on the input files and depending on the compromise between compression ratio and execution times that is desired, I must use various algorithms, including zstd and xz, but also bzip2, bzip3 and even gzip. I use quite frequently gzip (executed after lrzip preprocessing) for some very large files, where it provides better compression at a given execution time, or faster execution at a given compression ratio, than zstd with any options. Of course for other kinds of files, zstd wins, but all the claims that zstd should be used for ALL applications are extremely wrong. Whenever you must compress or decompress frequently, you must test all available algorithms with various parameters, to determine what works best for you. For big files, something like lrzip preprocessing must also be tested, as it can change a lot the performance of a compression algorithm.
saghm 8mo ago
Early on the article mentions that xz have zstd have gotten more popular than bzip, and my admitted naive understanding is that they're considered to have better tradeoffs in teems of collision compression time and overall space saved by compression. The performance section heavily discusses encoding performance of gzip and bzip, but unless I'm missing something, the only references to xz or zstd in that section are briefly handwaving about the decoding times probably being similar. My impression is that this article has a lot of technical insight into how bzip compares to gzip, but it fails actually account for the real cause of the diminished popularity of bzip in favor of the non-gzip alternatives that it admits are the more popular choices in recent years.
- 0cf8612b2e1e 8mo ago
  There was an analysis which argued that zstd is pareto optimal. https://insanity.industries/post/pareto-optimal-compression/ https://insanity.industries/post/pareto-optimal-compression/
  - lucb1e 8mo ago
    > > Pareto optimality is a situation where no […] preference criterion can be better off without making at least one […] preference criterion worse off […]. (Omissions theirs.) Wasn't that zstandard's stated goal? Not very surprising that it has this property, also considering it's much newer (2015) than the established tools like gzip (1992), bzip2 (1996), LZMA as used by xz utils (1999) Edit: https://github.com/facebook/zstd/blob/4856a00164c1d7b947bd384b7e9348b3a177d841/README.md https://github.com/facebook/zstd/blob/4856a00164c1d7b947bd38... initial commit indeed states it's meant to have good ratios and good (de)compression speeds as compared to other tools, without sacrificing one for the other (»"Standard" translates into everyday situations which neither look for highest possible ratio (which LZMA and ZPAQ cover) nor extreme speeds (which LZ4 covers).«). So Pareto by design, just not using that name
    - bonzini 8mo ago
      > meant to have good ratios and good (de)compression speeds as compared to other tools That does not mean it's Pareto optimal; Pareto-optimality forms a curve and while zstd, LZMA, LZ4, ZPAQ all want to be as close as possible to the curve, they focus on different parts of it. In particular zstd tries to stay on the middle part of the curve, while LZMA and LZ4 focus on opposite sides ___.--- higher throughput / LZ4 / zStd | ; LZMA | | ZPAQ lower size Also, the Pareto curve is not necessarily known in advance. All you can do is add more and more algorithms or tweaks to understand what it looks like. For example, this blog post [https://insanity.industries/post/pareto-optimal-compression/ https://insanity.industries/post/pareto-optimal-compression/] shows that prior to zstd, bzip2 and gzip2 were both pretty much Pareto optimal in the same area for ratio vs. compression speed. LZMA at low settings was a bit better but much slower. There was a huge gap between LZMA and LZ4, and bzip2/gzip filled it as best as they could. The same blog post shows that zstd is an absolute speed demon at decompression; while not all zstd settings are Pareto optimal when looking at size vs compression speed (in particular LZMA wins at higher compression ratios, and even considering zstd only there's hardly a reason to use levels 11-15), zstd is pretty much Pareto optimal at all settings when looking at size vs. decompression speed. On the other hand at intermediate settings zstd is faster and produces smaller files than gzip, which therefore is not Pareto optimal (anymore).
hexxagone 8mo ago
Notice that bzip3 has close to nothing to do with bzip2. It is a different BWT implementation with a different entropy codec, from a different author (as noted in the GitHub description "better and stronger spiritual successor to BZip2").
- lucb1e 8mo ago
  Was the name used with permission? Even if not trademarked (because open source freedom woohoo), it's a bit weird to release, say, Windows 12 without permission from the authors of Windows 11 I tried looking it up myself but it doesn't say in the readme or doc/ folder, there is no mention of any of the Bzip2 authors, and there is no website listed so I presume this Github page is canonical
  - tosti 8mo ago
    Free software doesn't have a lot of trademarks, with the notable exception of Linux. Also, the name is the algorithm. Bzip2 has versions and bzip3 is something else which has its own updated versions. Programs that implement a single algorithm often follow this pattern.
    - lucb1e 8mo ago
      > Free software doesn't have a lot of trademarks Hence my saying "not trademarked (because open source freedom woohoo)". I understand bzip3 is legal, but my question is whether it's a dick move > the name is the algorithm. Bzip2 has versions and bzip3 is something else I don't understand this. If bzip3 is "something else" compared to bzip2 then how can both be named after the algorithms they use? Nothing in bzip2's algorithm is related to the digit 2. If it used, say, powers of 2 centrally and bzip3 used pi for something, then the naming could make sense since it's indeed not a version number, but I don't remotely see this argument in these algorithms Looking on Wikipedia and bzip2's website, there is no explanation of the name, and yesterday I read somewhere (either in the OP or in another comment) that it would stand for "better zip". It has nothing to do with PKZIP though. If they had both implemented pkzip's format, then a "spiritual successor" to bzip2 would be something like uzip, uzip2 (replacing "better" with "ultimate"), bbzip2, or any of a million other options, but that's not the pattern they chose to follow How is this not just taking their established name for your own project with the version number +1?
fl0ki 8mo ago
This seems as good a thread as any to mention that the gzhttp package in klauspost/compress for Go now supports zstd on both server handlers and client transports. Strangely this was added in a patch version instead of a minor version despite both expanding the API surface and changing default behavior. https://github.com/klauspost/compress/releases/tag/v1.18.4 https://github.com/klauspost/compress/releases/tag/v1.18.4
- klauspost 8mo ago
  About the versioning, glad you spotted it anyway. There isn't as much use of the gzhttp package compared to the other ones, so the bar is a bit higher for that one. Also making good progress on getting a slimmer version of zstd into the stdlib and improving the stdlib deflate.
  - fl0ki 8mo ago
    Yeah, I make it a habit to read the changelogs of every update to every direct dependency. I was anticipating this change for years, thanks for doing it!
  - terrelln 8mo ago
    > Also making good progress on getting a slimmer version of zstd into the stdlib Awesome! Please let me know if there is anything I can do to help
    - klauspost 8mo ago
      https://github.com/klauspost/stdgozstd https://github.com/klauspost/stdgozstd
pella 8mo ago
imho: the future is a specialized compressor optimized for your specific format. ( https://openzl.org/ https://openzl.org/ , ... )
- srean 8mo ago
  That is an interesting link. Does gmail use a special codec for storing emails ?
  - duskwuff 8mo ago
    The biggest savings for a service like GMail are going to be based around deduplication - e.g. if you can recognize that a newsletter went out to a thousand subscribers and store those all as deltas from a "canonical" copy - congratulations, that's >1000:1 compression, better than you could achieve with any general-purpose compression. Similarly, if you can recognize that an email is an Amazon shipping confirmation or a Facebook message notification or some other commonly repeated "form letter", you can achieve huge savings by factoring out all the common elements in them, like images or stylesheets.
    - dataflow 8mo ago
      I kind of doubt they would do this to be honest. Every near-copy of a message is going to have small differences in at least the envelope (not sure if encoding differences are also possible depending on the server), and possibly going to be under different guarantees or jurisdictions. And it would just take one mistake to screw things up and leak data from one person to another. All for saving a few gigabytes over an account's lifetime. Doesn't really seem worth it, does it?
      - srean 8mo ago
        That's why a base and a delta. Whereas PP was talking about general compression algorithm, my question was different. In line with the original comment, I was asking about specialized "codecs" for gmail. Humans do not read the same email many times. That makes it a good target for compression. I believe machines do read the same email many times, but that could be architected around.
Grom_PE 8mo ago
PPMd (of 7-Zip) would beat BZip2 for compressing plain text data.
comet_browser 8mo ago
[flagged]
- nvme0n1p1 8mo ago
  ai slop account
idoubtit 8mo ago
My experience does not match theirs when compressing text and code: > bzip might be suboptimal as a general-purpose compression format, but it’s great for text and code. One might even say the b in bzip stands for “best”. I've just checked again with a 1GB SQL file. `bzip2 -9` shrinks it to 83MB. `zstd -19 --long` to 52MB. Others have compressed the Linux kernel and found that bzip2's is about 15% larger than zstd's.
- cogman10 8mo ago
  bzip is old and slow. It was long surpassed by lzma and zstd. But back in roughly the 00s, it was the best standard for compression, because the competition was DEFLATE/gzip.
  - duskwuff 8mo ago
    Also potentially relevant: in the 00s, the performance gap between gzip and bzip2 wasn't quite as wide - gzip has benefited far more from modern CPU optimizations - and slow networks / small disks made a higher compression ratio more valuable.
  - yyyk 8mo ago
    Even then, there were better options in the Windows world (RAR/ACE/etc.). Also, bzip2 was considered slow even when it was new.
    - thesz 8mo ago
      RAR/ACE/etc used continuous compression - all files were concatenated and compressed as if they were one single large file. Much like what is done with .tar.bz. Bzip on Windows did not do that, there was no equivalent of .tar.bz2 on Windows. You can bzip2 -9 files in some source code directory and tar these .bz2 files. This would be more or less equivalent to creating ZIP archive with BWT compression method. Then you can compare result with tar-ing the same source directory and bzip2 -9 the resulting .tar. Then you can compare. The continuous mode in RAR was something back then, exactly because RAR had long LZ77 window and compressed files as continuous stream.
deleted 8mo ago
[deleted]
cobbzilla 8mo ago
My first exposure to bzip: The first Linux kernels I ever compiled & built myself (iirc ~v2.0.x), I packed as .tar.bz2 images. Ah the memories. Yes, there are better compression options today.
vintermann 8mo ago
If you're implementing it for Computercraft anyway, there's no reason to stick to the standard. It's well known that bzip2 has a couple of extra steps which don't improve compression ratio at all. I suggest implementing Scott's Bijective Burrows-Wheeler variant on bits rather than bytes, and do bijective run-length encoding of the resulting string. It's not exactly on the "pareto frontier", but it's fun!
thesz 8mo ago
BWT is a prediction by partial match (PPM) in disguise. Consider "bananarama": "abananaram" "amabananar" "ananaramab" "anaramaban" "aramabanan" "bananarama" "mabananara" "nanaramaba" "naramabana" "ramabanana" The last symbols on each line get context from first symbols of the same line. It is so due to rotation. But, due to sorting, contexts are not contiguous for the (last) character predicted and long dependencies are broken. Because of broken long dependencies, it is why MTF, which implicitly transforms direct symbols statistics into something like Zipfian [1] statistics, does encode BWT's output well. [1] https://en.wikipedia.org/wiki/Zipf%27s_law https://en.wikipedia.org/wiki/Zipf%27s_law Given that, author may find PPM*-based compressors to be more compression-wise performant. Large Text Compression Benchmark [2] tells us exactly that: some "durilka-bububu" compressor that uses PPM fares better than BWT, almost by third.
eichin 8mo ago
Interesting detail on the algorithm but seems to completely miss that if you care about non-streaming performance, there are parallel versions of xz and gzip (pxzip encodes compatible metadata about the breakup points so that while xz can still decompress it, pxzip can use as many cores as you let it have instead.) Great for disk-image OS installers (the reason I was benchmarking it in the first place - but this was about 5 years back, I don't know if those have gotten upstreamed...)
- shawn_w 8mo ago
  There's also a parallel version of bzip2, pbzip2. https://man.archlinux.org/man/pbzip2.1.en https://man.archlinux.org/man/pbzip2.1.en And zstd is multi threaded from the beginning.
ryan14975 8mo ago
[flagged]
- foltik 8mo ago
  LLM slop, and also incorrect. 900kb is not “buffering the entire input,” it’s just a slightly larger buffer size. Both are streaming algorithms.
  - jiggawatts 8mo ago
    It's a bot/shill account advertising "zip as a service". I wish I was kidding.
jiggawatts 8mo ago
I recently did some works on genomics / bioinformatics, where terabyte-sized text datasets are common and often compressed and decompressed multiple times in some workloads. This often becomes the serial bottleneck we all know and love from Amdahl's law. I ran a bunch of benchmarks, and found that the only thing that mattered was if a particular tool or format supported parallel compression and/or parallel decompression. Nothing else was even close as a relevant factor. If you're developing software for processing even potentially large files and you're using a format that is inherently serial, you've made a mistake. You're wasting 99.5% of a modern server's capacity, and soon that'll be 99.9%. It really, really doesn't matter if one format is 5% faster or 5% bigger or whatever if you're throwing away a factor of 200 to 1,000 speedup that could be achieved through parallelism! Or conversely, the ability to throw up to 1,000x the compute at improving the compression ratio in the available wall clock time. Checksumming, compression, decompression, encryption, and decryption over bulk data must all switch to fully parallel codecs, now. Not next year, next decade, or the year after we have 1,000+ core virtual machines in the cloud available on a whim. Oh wait, we do already: https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/memory-optimized/mdsv3-vhm-series?tabs=sizebasic#host-specifications https://learn.microsoft.com/en-us/azure/virtual-machines/siz... Not to mention: 200-400 Gbps NICs are standard now in all HPC VMs and starting to become commonplace in ordinary "database optimised" cloud virtual machines. Similarly, local and remote SSD-backed volumes that can read and write at speeds north of 10 GB/s. There are very few (any?) compression algorithms that can keep up with a single TCP/IP stream on a data centre NIC or single file read from a modern SSD unless using parallelism. Similarly, most CPUs struggle to perform even just SHA or AES at those speeds on a single core.
- hexxagone 8mo ago
  Then you should take a look at https://github.com/flanglet/kanzi-cpp https://github.com/flanglet/kanzi-cpp: it is optimized for fast roundtrips, multi-threaded by design and produces a seekable bitstream.
roytam87 8mo ago
I wonder why people think there is only 1 bzip (which seems to be bzip2), when the original bzip exists: https://unix.stackexchange.com/a/125803 https://unix.stackexchange.com/a/125803 and also bzip3: https://github.com/iczelia/bzip3 https://github.com/iczelia/bzip3

An ode to bzip

170 points by signa11 8mo ago

elophanto_agent 8mo ago

[dead]

kergonath 8mo ago
Bzip2 is slow. That’s the main issue. Gzip is good enough and much faster. Also, the fact that you cannot get a valid bzip2 file by cat-ing 2 compressed files is not a deal breaker, but it is annoying.
- saidnooneever 8mo ago
  the catting issue might be more an implementation of bzip program problem than algorithm (it could expect an array of compressed files). that would only be impossible if the program cannot reason about the length of data from file header, which again is technically not something about compression algo but rather file format its carried through. that being said, speed is important for compression so for systems like webservers etc its an easy sell ofc. very strong point (and smarter implementation in programs) for gzip
  - nine_k 8mo ago
    Bzip2 is great for files that are compressed once, get decompressed many times, and the size is important. A good example is a software release.
    - pocksuppet 8mo ago
      So is xz, or zstd, and the files are smaller. bzip2 disappeared from software releases when xz was widely available. gzip often remains, as the most compatible option, the FAT32 of compression algorithms.
    - lucb1e 8mo ago
      Huh? Only if it gets decompressed few times I would say, because it's so extremely slow at it Like a software installation that you do one time. I'd not even want it for updates if I need to update large data files. The only purpose I'd see is the first-time install where users are okay waiting a while, and small code patches that are distributed to a lot of people (Or indeed if size is important, but then again bzip2 only shines if it's text-like. I don't really find this niche knowledge for a few % optimization worth teaching tbh. Better teach general principles so people can find a fitting solution if they ever find themselves under specific constraints like OP)
  - joecool1029 8mo ago
    > the catting issue might be more an implementation of bzip program problem than algorithm (it could expect an array of compressed files). that would only be impossible if the program cannot reason about the length of data from file header, which again is technically not something about compression algo but rather file format its carried through. Long comment to just say: ‘I have no idea about what I’m writing about’ These compression algorithms do not have anything to do with filesystem structure. Anyway the reason you can’t cat together parts of bzip2 but you can with zstd (and gzip) is because zstd does everything in frames and everything in those frames can be decompressed separately (so you can seek and decompress parts). Bzip2 doesn’t do that. So like, another place bzip2 sucks ass is working with large archives because you need to seek the entire archive before you can decompress it and it makes situations without parity data way more likely to cause dataloss of the whole archive. Really, don’t use it unless you have a super specific use case and know the tradeoffs, for the average person it was great when we would spend the time compressing to save the time sending over dialup.
    - prerok 8mo ago
      So... it's actually a reasonable objection over bzip2? I mean, you explained why it does not work with bzip2. I think their argument is sound and it makes using bzip2 less useful in certain situations. I was once saved in resolving a problem we had when I figured out that concatening gzipped files just works out of the box. If not, it would have meant a bit more code, lots of additional testing, etc.
      - saidnooneever 7mo ago
        totally agree with the statement though i feel its not an objection over bzip 2 rather than how it was implemented in programs that apply it. but i'm not really 100% since admittedly i did not personally reverse engineer bzip capable programs to see the current state of afairs. I am simply going by descriptions posted in comments and general system knowlesge. how to compress data has little to no relation to how this compression can be implemented in programs. How its implemented, will reflect on how the quality of the algorithm is perceived, becaus e the two are not seperate from a user perspective.
- nine_k 8mo ago
  Gzip is woefully old. Its only redeeming value is that it's already built into some old tools. Otherwise, use zstd, which is better and faster, both at compression and decompression. There's no reason to use gzip in anything new, except for backwards compatibility with something old.
  - kergonath 8mo ago
    > Otherwise, use zstd, which is better and faster Yes, I do. Zstd is my preferred solution nowadays. But gzip is not going anywhere as a fallback because there is a surprisingly high number of computers without a working libzstd.
  - duskwuff 8mo ago
    One other redeeming quality that gzip/deflate does have is that its low memory requirements (~32 KB per stream). If you're running on an embedded device, or if you're serving a ton of compressed streams at the same time, this can be a meaningful benefit.
  - adrian_b 8mo ago
    The claim that zstd is "better and faster", without additional qualifications, is false and misleading. Indeed for many simple test cases zstd is both better and faster. Despite that, it is possible to find input files for which zstd is either worse or slower than gzip, for any zstd options. I have tested zstd a lot, but eventually I have chosen to use lrzip+gzip for certain very large files (tens of GB file size) with which I am working, because zstd was always either worse or slower. (For those files, at the same compression ratio and on a 6-year old PC, lrzip+gzip has a compression speed always greater than 100 MB/s, while zstd only of 30 to 40 MB/s.) There are also other applications, where I do use zstd, but no compressing program is better than all the others in ALL applications.
- stefan_ 8mo ago
  bzip and gzip are both horrible, terribly slow. Wherever I see "gz" or "bz" I immediately rip that nonsense out for zstd. There is such a thing as a right choice, and zstd is it every time.
  - laurencerowe 8mo ago
    lz4 can still be the right choice when decompression speed matters. It's almost twice as fast at decompression with similar compression ratios to zstd's fast setting. https://github.com/facebook/zstd?tab=readme-ov-file#benchmarks https://github.com/facebook/zstd?tab=readme-ov-file#benchmar...
  - anthk 8mo ago
    pigz it's damn fast on compressing. Also, a Vax with NetBSD can run gzip. So here is it. Go try these new fancy formats on a Vax, I dare you. And, yes, I prefer LZMA over the obsolete Bzip2 any day, but GZIP it's like the ZIP of free formats modulo packaging, which it's the job of TAR.
  - kergonath 8mo ago
    > Wherever I see "gz" or "bz" That should not happen too often, considering that IIRC bzip lasted only a couple of months before being replaced by bzip2.
- sedatk 8mo ago
  > the fact that you cannot get a valid bzip2 file by cat-ing 2 compressed files TIL. Now that's why gzip has a file header! But, tar.gz compresses even better, that's probably why it hasn't caught on.
  - pocksuppet 8mo ago
    tar packs multiple files into one. If you concatenate two gzipped files and unzip them, you just get a concatenated file.
    - sedatk 8mo ago
      Ah okay, I thought gzip would support decompressing multiple files that way.
      - kergonath 8mo ago
        How it works is, if you have two files foo.gz and bar.gz, and cat foo.gz bar.gz > foobar.gz, then foobar.gz is a valid gzip file and uncompresses to a single file with the contents of foo and bar. It’s handy because it is very easy to just append stuff at the end of a compressed file without having to uncompress-append-recompress. It is a bit niche but I have a couple of use cases where it makes everything simpler.
      - bmacho 8mo ago
        tar supports that types of concatenation, so you can concatenate tar.gz files, and unpack them all into separate files
        sedatk 8mo ago
        I know, but I've been always confused why a gzip file would have a filename field in its header if it's supposed to contain only one file. Obviously it's good to keep a backup of original filename somewhere, but it's confusing nonetheless.
- duskwuff 8mo ago
  bzip2 is particularly slow because the transform it depends on (BWT2) is "intrinsically slow" - it depends on cache-unfriendly operations with long dependency chains, preventing the CPU from extracting any parallelism: https://cbloomrants.blogspot.com/2021/03/faster-inverse-bwt.html https://cbloomrants.blogspot.com/2021/03/faster-inverse-bwt....
singpolyma3 8mo ago
Neither has been good enough for years.

joecool1029 8mo ago

Just use zstd unless you absolutely need to save a tiny bit more space. bzip2 and xz are extremely slow to compress.

silisili 8mo ago
I'd argue it's more workload dependent, and everything is a tradeoff. In my own testing of compressing internal generic json blobs, I found brotli a clear winner when comparing space and time. If I want higher compatibility and fast speeds, I'd probably just reach for gzip. zstd is good for many use cases, too, perhaps even most...but I think just telling everyone to always use it isn't necessarily the best advice.
- joecool1029 8mo ago
  > If I want higher compatibility and fast speeds, I'd probably just reach for gzip. It’s slower and compresses less than zstd. gzip should only be reached for as a compatibility option, that’s the only place it wins, it’s everywhere. EDIT: If you must use it, use the modern implementation, https://www.zlib.net/pigz/ https://www.zlib.net/pigz/
  - adrian_b 8mo ago
    Any claims about compressing programs are extremely data-dependent so any general claims will be false for certain test cases. I do a lot of data compression and decompression, and I would have liked a lot to find a magic algorithm that works better than all others, to simplify my work. After extensive tests I have found no such algorithm. Depending on the input files and depending on the compromise between compression ratio and execution times that is desired, I must use various algorithms, including zstd and xz, but also bzip2, bzip3 and even gzip. I use quite frequently gzip (executed after lrzip preprocessing) for some very large files, where it provides better compression at a given execution time, or faster execution at a given compression ratio, than zstd with any options. Of course for other kinds of files, zstd wins, but all the claims that zstd should be used for ALL applications are extremely wrong. Whenever you must compress or decompress frequently, you must test all available algorithms with various parameters, to determine what works best for you. For big files, something like lrzip preprocessing must also be tested, as it can change a lot the performance of a compression algorithm.

saghm 8mo ago

Early on the article mentions that xz have zstd have gotten more popular than bzip, and my admitted naive understanding is that they're considered to have better tradeoffs in teems of collision compression time and overall space saved by compression. The performance section heavily discusses encoding performance of gzip and bzip, but unless I'm missing something, the only references to xz or zstd in that section are briefly handwaving about the decoding times probably being similar. My impression is that this article has a lot of technical insight into how bzip compares to gzip, but it fails actually account for the real cause of the diminished popularity of bzip in favor of the non-gzip alternatives that it admits are the more popular choices in recent years.

0cf8612b2e1e 8mo ago
There was an analysis which argued that zstd is pareto optimal. https://insanity.industries/post/pareto-optimal-compression/ https://insanity.industries/post/pareto-optimal-compression/
- lucb1e 8mo ago
  > > Pareto optimality is a situation where no […] preference criterion can be better off without making at least one […] preference criterion worse off […]. (Omissions theirs.) Wasn't that zstandard's stated goal? Not very surprising that it has this property, also considering it's much newer (2015) than the established tools like gzip (1992), bzip2 (1996), LZMA as used by xz utils (1999) Edit: https://github.com/facebook/zstd/blob/4856a00164c1d7b947bd384b7e9348b3a177d841/README.md https://github.com/facebook/zstd/blob/4856a00164c1d7b947bd38... initial commit indeed states it's meant to have good ratios and good (de)compression speeds as compared to other tools, without sacrificing one for the other (»"Standard" translates into everyday situations which neither look for highest possible ratio (which LZMA and ZPAQ cover) nor extreme speeds (which LZ4 covers).«). So Pareto by design, just not using that name
  - bonzini 8mo ago
    > meant to have good ratios and good (de)compression speeds as compared to other tools That does not mean it's Pareto optimal; Pareto-optimality forms a curve and while zstd, LZMA, LZ4, ZPAQ all want to be as close as possible to the curve, they focus on different parts of it. In particular zstd tries to stay on the middle part of the curve, while LZMA and LZ4 focus on opposite sides ___.--- higher throughput / LZ4 / zStd | ; LZMA | | ZPAQ lower size Also, the Pareto curve is not necessarily known in advance. All you can do is add more and more algorithms or tweaks to understand what it looks like. For example, this blog post [https://insanity.industries/post/pareto-optimal-compression/ https://insanity.industries/post/pareto-optimal-compression/] shows that prior to zstd, bzip2 and gzip2 were both pretty much Pareto optimal in the same area for ratio vs. compression speed. LZMA at low settings was a bit better but much slower. There was a huge gap between LZMA and LZ4, and bzip2/gzip filled it as best as they could. The same blog post shows that zstd is an absolute speed demon at decompression; while not all zstd settings are Pareto optimal when looking at size vs compression speed (in particular LZMA wins at higher compression ratios, and even considering zstd only there's hardly a reason to use levels 11-15), zstd is pretty much Pareto optimal at all settings when looking at size vs. decompression speed. On the other hand at intermediate settings zstd is faster and produces smaller files than gzip, which therefore is not Pareto optimal (anymore).

hexxagone 8mo ago

Notice that bzip3 has close to nothing to do with bzip2. It is a different BWT implementation with a different entropy codec, from a different author (as noted in the GitHub description "better and stronger spiritual successor to BZip2").

lucb1e 8mo ago
Was the name used with permission? Even if not trademarked (because open source freedom woohoo), it's a bit weird to release, say, Windows 12 without permission from the authors of Windows 11 I tried looking it up myself but it doesn't say in the readme or doc/ folder, there is no mention of any of the Bzip2 authors, and there is no website listed so I presume this Github page is canonical
- tosti 8mo ago
  Free software doesn't have a lot of trademarks, with the notable exception of Linux. Also, the name is the algorithm. Bzip2 has versions and bzip3 is something else which has its own updated versions. Programs that implement a single algorithm often follow this pattern.
  - lucb1e 8mo ago
    > Free software doesn't have a lot of trademarks Hence my saying "not trademarked (because open source freedom woohoo)". I understand bzip3 is legal, but my question is whether it's a dick move > the name is the algorithm. Bzip2 has versions and bzip3 is something else I don't understand this. If bzip3 is "something else" compared to bzip2 then how can both be named after the algorithms they use? Nothing in bzip2's algorithm is related to the digit 2. If it used, say, powers of 2 centrally and bzip3 used pi for something, then the naming could make sense since it's indeed not a version number, but I don't remotely see this argument in these algorithms Looking on Wikipedia and bzip2's website, there is no explanation of the name, and yesterday I read somewhere (either in the OP or in another comment) that it would stand for "better zip". It has nothing to do with PKZIP though. If they had both implemented pkzip's format, then a "spiritual successor" to bzip2 would be something like uzip, uzip2 (replacing "better" with "ultimate"), bbzip2, or any of a million other options, but that's not the pattern they chose to follow How is this not just taking their established name for your own project with the version number +1?

fl0ki 8mo ago

This seems as good a thread as any to mention that the gzhttp package in klauspost/compress for Go now supports zstd on both server handlers and client transports. Strangely this was added in a patch version instead of a minor version despite both expanding the API surface and changing default behavior. https://github.com/klauspost/compress/releases/tag/v1.18.4 https://github.com/klauspost/compress/releases/tag/v1.18.4

klauspost 8mo ago
About the versioning, glad you spotted it anyway. There isn't as much use of the gzhttp package compared to the other ones, so the bar is a bit higher for that one. Also making good progress on getting a slimmer version of zstd into the stdlib and improving the stdlib deflate.
- fl0ki 8mo ago
  Yeah, I make it a habit to read the changelogs of every update to every direct dependency. I was anticipating this change for years, thanks for doing it!
- terrelln 8mo ago
  > Also making good progress on getting a slimmer version of zstd into the stdlib Awesome! Please let me know if there is anything I can do to help
  - klauspost 8mo ago
    https://github.com/klauspost/stdgozstd https://github.com/klauspost/stdgozstd

pella 8mo ago

imho: the future is a specialized compressor optimized for your specific format. ( https://openzl.org/ https://openzl.org/ , ... )

srean 8mo ago
That is an interesting link. Does gmail use a special codec for storing emails ?
- duskwuff 8mo ago
  The biggest savings for a service like GMail are going to be based around deduplication - e.g. if you can recognize that a newsletter went out to a thousand subscribers and store those all as deltas from a "canonical" copy - congratulations, that's >1000:1 compression, better than you could achieve with any general-purpose compression. Similarly, if you can recognize that an email is an Amazon shipping confirmation or a Facebook message notification or some other commonly repeated "form letter", you can achieve huge savings by factoring out all the common elements in them, like images or stylesheets.
  - dataflow 8mo ago
    I kind of doubt they would do this to be honest. Every near-copy of a message is going to have small differences in at least the envelope (not sure if encoding differences are also possible depending on the server), and possibly going to be under different guarantees or jurisdictions. And it would just take one mistake to screw things up and leak data from one person to another. All for saving a few gigabytes over an account's lifetime. Doesn't really seem worth it, does it?
    - srean 8mo ago
      That's why a base and a delta. Whereas PP was talking about general compression algorithm, my question was different. In line with the original comment, I was asking about specialized "codecs" for gmail. Humans do not read the same email many times. That makes it a good target for compression. I believe machines do read the same email many times, but that could be architected around.

Grom_PE 8mo ago

PPMd (of 7-Zip) would beat BZip2 for compressing plain text data.

comet_browser 8mo ago

[flagged]

nvme0n1p1 8mo ago
ai slop account

idoubtit 8mo ago

My experience does not match theirs when compressing text and code: > bzip might be suboptimal as a general-purpose compression format, but it’s great for text and code. One might even say the b in bzip stands for “best”. I've just checked again with a 1GB SQL file. `bzip2 -9` shrinks it to 83MB. `zstd -19 --long` to 52MB. Others have compressed the Linux kernel and found that bzip2's is about 15% larger than zstd's.

cogman10 8mo ago
bzip is old and slow. It was long surpassed by lzma and zstd. But back in roughly the 00s, it was the best standard for compression, because the competition was DEFLATE/gzip.
- duskwuff 8mo ago
  Also potentially relevant: in the 00s, the performance gap between gzip and bzip2 wasn't quite as wide - gzip has benefited far more from modern CPU optimizations - and slow networks / small disks made a higher compression ratio more valuable.
- yyyk 8mo ago
  Even then, there were better options in the Windows world (RAR/ACE/etc.). Also, bzip2 was considered slow even when it was new.
  - thesz 8mo ago
    RAR/ACE/etc used continuous compression - all files were concatenated and compressed as if they were one single large file. Much like what is done with .tar.bz. Bzip on Windows did not do that, there was no equivalent of .tar.bz2 on Windows. You can bzip2 -9 files in some source code directory and tar these .bz2 files. This would be more or less equivalent to creating ZIP archive with BWT compression method. Then you can compare result with tar-ing the same source directory and bzip2 -9 the resulting .tar. Then you can compare. The continuous mode in RAR was something back then, exactly because RAR had long LZ77 window and compressed files as continuous stream.

deleted 8mo ago

[deleted]

cobbzilla 8mo ago

My first exposure to bzip: The first Linux kernels I ever compiled & built myself (iirc ~v2.0.x), I packed as .tar.bz2 images. Ah the memories. Yes, there are better compression options today.

vintermann 8mo ago

If you're implementing it for Computercraft anyway, there's no reason to stick to the standard. It's well known that bzip2 has a couple of extra steps which don't improve compression ratio at all. I suggest implementing Scott's Bijective Burrows-Wheeler variant on bits rather than bytes, and do bijective run-length encoding of the resulting string. It's not exactly on the "pareto frontier", but it's fun!

thesz 8mo ago

BWT is a prediction by partial match (PPM) in disguise. Consider "bananarama": "abananaram" "amabananar" "ananaramab" "anaramaban" "aramabanan" "bananarama" "mabananara" "nanaramaba" "naramabana" "ramabanana" The last symbols on each line get context from first symbols of the same line. It is so due to rotation. But, due to sorting, contexts are not contiguous for the (last) character predicted and long dependencies are broken. Because of broken long dependencies, it is why MTF, which implicitly transforms direct symbols statistics into something like Zipfian [1] statistics, does encode BWT's output well. [1] https://en.wikipedia.org/wiki/Zipf%27s_law https://en.wikipedia.org/wiki/Zipf%27s_law Given that, author may find PPM*-based compressors to be more compression-wise performant. Large Text Compression Benchmark [2] tells us exactly that: some "durilka-bububu" compressor that uses PPM fares better than BWT, almost by third.

eichin 8mo ago

Interesting detail on the algorithm but seems to completely miss that if you care about non-streaming performance, there are parallel versions of xz and gzip (pxzip encodes compatible metadata about the breakup points so that while xz can still decompress it, pxzip can use as many cores as you let it have instead.) Great for disk-image OS installers (the reason I was benchmarking it in the first place - but this was about 5 years back, I don't know if those have gotten upstreamed...)

shawn_w 8mo ago
There's also a parallel version of bzip2, pbzip2. https://man.archlinux.org/man/pbzip2.1.en https://man.archlinux.org/man/pbzip2.1.en And zstd is multi threaded from the beginning.

ryan14975 8mo ago

[flagged]

foltik 8mo ago
LLM slop, and also incorrect. 900kb is not “buffering the entire input,” it’s just a slightly larger buffer size. Both are streaming algorithms.
- jiggawatts 8mo ago
  It's a bot/shill account advertising "zip as a service". I wish I was kidding.

jiggawatts 8mo ago

I recently did some works on genomics / bioinformatics, where terabyte-sized text datasets are common and often compressed and decompressed multiple times in some workloads. This often becomes the serial bottleneck we all know and love from Amdahl's law. I ran a bunch of benchmarks, and found that the only thing that mattered was if a particular tool or format supported parallel compression and/or parallel decompression. Nothing else was even close as a relevant factor. If you're developing software for processing even potentially large files and you're using a format that is inherently serial, you've made a mistake. You're wasting 99.5% of a modern server's capacity, and soon that'll be 99.9%. It really, really doesn't matter if one format is 5% faster or 5% bigger or whatever if you're throwing away a factor of 200 to 1,000 speedup that could be achieved through parallelism! Or conversely, the ability to throw up to 1,000x the compute at improving the compression ratio in the available wall clock time. Checksumming, compression, decompression, encryption, and decryption over bulk data must all switch to fully parallel codecs, now. Not next year, next decade, or the year after we have 1,000+ core virtual machines in the cloud available on a whim. Oh wait, we do already: https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/memory-optimized/mdsv3-vhm-series?tabs=sizebasic#host-specifications https://learn.microsoft.com/en-us/azure/virtual-machines/siz... Not to mention: 200-400 Gbps NICs are standard now in all HPC VMs and starting to become commonplace in ordinary "database optimised" cloud virtual machines. Similarly, local and remote SSD-backed volumes that can read and write at speeds north of 10 GB/s. There are very few (any?) compression algorithms that can keep up with a single TCP/IP stream on a data centre NIC or single file read from a modern SSD unless using parallelism. Similarly, most CPUs struggle to perform even just SHA or AES at those speeds on a single core.

hexxagone 8mo ago
Then you should take a look at https://github.com/flanglet/kanzi-cpp https://github.com/flanglet/kanzi-cpp: it is optimized for fast roundtrips, multi-threaded by design and produces a seekable bitstream.

roytam87 8mo ago

I wonder why people think there is only 1 bzip (which seems to be bzip2), when the original bzip exists: https://unix.stackexchange.com/a/125803 https://unix.stackexchange.com/a/125803 and also bzip3: https://github.com/iczelia/bzip3 https://github.com/iczelia/bzip3