8 ms·

Zstandard Worked Example

175 points by stargrave 4y ago

londons_explore 4y ago
This post talksabout literals still being encoded with huffman tables and stuff. However, it appears this must be optional, because short strings compressed with zstd pass through 'unencoded', apart from a header and footer added: $ echo "hello world" | zstd | hexdump -C 00000000 28 b5 2f fd 04 58 61 00 00 68 65 6c 6c 6f 20 77 |(./..Xa..hello w| 00000010 6f 72 6c 64 0a 8c 6d 7d 20 |orld..m} | As an aside, I'm fairly disappointed that zstd didn't use a single byte header for uncompressible strings, so that they could guarantee that the compressed data will never be more than 1 byte larger than the source data. That property is very useful where lots of tiny strings are being compressed, such as in a database.
- wolf550e 4y ago
  This is an uncompressed block. For database records or log lines, you should use dictionary trained on likely data. Then short records compress well. See what facebook does with zstd (they employ its author): https://engineering.fb.com/2018/12/19/core-data/zstandard/ https://engineering.fb.com/2018/12/19/core-data/zstandard/
- pcwalton 4y ago
  The first 4 bytes are the magic number and the last 4 bytes are the checksum [1] which you could always just chop off if you wanted (it's legal to omit the checksum, see the spec). That would get the total overhead down to 5 bytes. [1]: https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#zstandard-frames https://github.com/facebook/zstd/blob/dev/doc/zstd_compressi...
- terrelln 4y ago
  Zstd has a 4-byte magic number, which is used to check if the data is zstd encoded. In addition to that, this example has a 2 byte frame header (including the decompressed size), a 3 byte block header, and a 4 byte checksum at the end (which can be disabled with `--no-check`). Zstd does have a mode that passes-through incompressible data, both for incompressible literals, and completely incompressible blocks (128 KB chunks). For small inputs, we recommend using dictionary compression. But even with dictionary compression, because of our header costs, you won't generally see benefits until your data is at least ~50 bytes. But YMMV depending on your data.
keyle 4y ago
Interesting, I've never heard of it before. I was aware that there were better standards than zip, however adoption was always an issue with licenses. If this is as open, is it purely an issue of wide adoption? We're stuck in the C++ paradigm (I'm sure there is a better name), where everyone agree this isn't "ideal", that there is better, but not widely adopted enough?
- learndeeply 4y ago
  Your questions are answered in the first sentence of the article. > Zstd or Zstandard (RFC 8478, first released in 2015) is a popular modern compression algorithm. It’s smaller (better compression ratio) and faster than the ubiquitous Zlib/Deflate (RFC 1950 and RFC 1951, first released in 1995).
- c0balt 4y ago
  The article already mentions zstd but for an IMO simple and accurate overview take a look at the recommendations from https://jolynch.github.io/posts/use_fast_data_algorithms/ https://jolynch.github.io/posts/use_fast_data_algorithms/ .
- wolf550e 4y ago
  Unless you have to use deflate/zlib/gzip/zip for compatibility or you're very constrained in amount of memory, you should always prefer zstd over deflate - it's always both faster and compresses better, so there is no tradeoff and using deflate is just wasteful. Google's brotli is a close competitor and got supported in Chrome first, so HTTP uses that more than it uses zstd. zstd has a very wide range of compression levels to tradeoff compression ratio for CPU time, much wider than the difference between gzip -1, gzip -6 (default) and gzip -9. zstd -3 is default. zstd -1 is very fast, zstd -19 compresses really well. zstd also has a long range mode, for archives that contain the same file a gigabyte away. gzip's window is only 32KB in size, so it can't notice any repetitions more than that far away. LZ4 by the same author as zstd aims at even faster compression, at the cost of worse compression ratio. It makes sense to use over fast network. Google snappy and LZO are the usual competitors. When you want better compression than zstd (at the cost of CPU time, both during compression and during decompression), use PPMD for natural language text or source code and use LZMA (xz) for other things.
dietrichepp 4y ago
Nice writeup! FSE, one of the parts of Zstandard, is a very elegant entropy coder. If you're following this section of the article, you might be wondering how it's encoded, since the encoding process is somewhat less obvious than for Huffman coding. What causes trouble is simply, "Once you've emitted a symbol, how do you know that the next symbol is in the range [BL, BL+2^NB)?" The answer is that the encoding is performed backwards, starting from the end of the input, and the table is constructed so that for any symbol emitted, you can find a previous symbol so that the current symbol is in the previous symbol's [BL, BL+2^NB) range. There's another writeup about FSE here, which goes into more depth about FSE (rather than talking about Zstandard in general): http://fastcompression.blogspot.com/2013/12/finite-state-entropy-new-breed-of.html http://fastcompression.blogspot.com/2013/12/finite-state-ent... My intuition tells me that there's probably a homomorphism from Huffman coding to FSE that preserves the encoded size exactly, but I haven't done the math to check.
- FullyFunctional 4y ago
  That smells an awful lot like arithmetic coding. Is it just a reformulation (genuinely curious)?
  - dietrichepp 4y ago
    I wouldn't describe it as a reformulation, but as a different solution to the same problem. The goal of entropy coders in general is to encode each symbol using -log P bits, where P is the probability of that symbol. Huffman, arithmetic encoding, range coding, and ANS / FSE are different systems that solve this problem. https://en.wikipedia.org/wiki/Entropy_coding https://en.wikipedia.org/wiki/Entropy_coding Huffman coding works by using an integer number of bits for each symbol and then creating a prefix-free code. This is computationally cheap but wastes some space. Arithmetic coding works using interval arithmetic. Each symbol corresponds to an interval, and you multiply each successive interval together, renormalizing after each step. This is close to optimal but computationally expensive. ANS / FSE work by using a state machine. A particular symbol can be coded using a variable number of bits, but that number is an integer. The average number of bits is close to -log P. Like Huffman coding, coding and decoding is fast--each symbol uses an integer number of bits, and you don't need to do any multiplication. Like Arithmetic coding, it is close to optimal in terms of space used. These techniques share one thing in common--you only need to record symbol frequency in order to recreate the codebook. In Huffman coding, you typically encode the symbol frequency log 2 (i.e. the symbol length). In FSE and arithmetic coding, you typically encode the symbol frequency as a dyadic rational. You use a deterministic process to recreate the exact same codebook given these frequencies. (Then the question is, "How do you encode the symbol frequency efficiency?" For Deflate, the answer is, funny enough, the symbol lengths for the Huffman codes are themselves encoded using Huffman codes! The codebook for that Huffman code is then coded using a fixed codebook.)
terrelln 4y ago
Awesome post, it's great to see someone diving into the details of Zstd! Maintainer of Zstd here, AMA
- Rafuino 4y ago
  Ohh, don't mind if I do! I'm working on CPU libraries to improve compression performance. Does zstd benefit today from ISAs like AVX-512, AVX-2, etc.? Do you foresee the need in the future to offload compression/decompression to an accelerator? Another Q, what hardware platform(s) do you use to test new versions of zstd for regressions/improvements? I see the Github page shows a consumer CPU in use, but what about server CPUs pushing maximum throughput with many threads/cores?
  - terrelln 4y ago
    > Does zstd benefit today from ISAs like AVX-512, AVX-2, etc.? Zstd benefits mostly from BMI(2), it takes advantage of shlx, shrx, and bsf during entropy (de)coding. We do use SSE2 instructions in our lazy match finder to filter matches based on a 1-byte hash, in a similar way that F14 and Swiss hash tables do. We also use vector loads/stores to do our copies during decompression. We don't currently benefit from AVX2. There are strategies to use AVX2 for Huffman & FSE compression, but they don't fit well with zstd's format, so we don't use them there. Additionally, our latest Huffman decoder is very competitive with the AVX2 decoder on real data. And a Huffman format that is fast for AVX2 is often slow for scalar decoding, so it is hard to opt into it unless you know you will only run on CPUs that support AVX2. Lastly, AVX2 entropy decoders rely on a fast gather instruction, which isn't available on AMD machines. > Do you foresee the need in the future to offload compression/decompression to an accelerator? Yes, there are trends in this direction. Intel QAT supports zlib hardware acceleration. The PS5 has accelerated decompression. I'm pretty sure Microsoft has also released a paper about a HW accelerated algorithm. Compression & decompression take up a significant portion of datacenter CPU, so it makes sense to hardware accelerate it. > what hardware platform(s) do you use to test new versions of zstd for regressions/improvements? We mostly test and optimize for x86-64 CPUs, on a mix of server and consumer CPUs. But, we also test on ARM devices to make sure we don't have major regressions. We've found that optimizing for x86-64 has done a good job of getting good baseline performance on other architectures, though there is certainly room left to improve.
userbinator 4y ago
LZ followed by entropy coding seems to be the general strategy of choice for many compression algorithms, so this design of Zstandard was quite familiar to me. The FSE reminded me of the Q/MQ/QM-coder algorithm which is used in some image formats (and is surprisingly simple yet powerful, although relatively slow.) Nonetheless, I like to learn a file format by studying a worked example at the bits and bytes level. I did the same when I was playing around with various image and video codecs (JPEG, GIF, MPEG-1/2, etc.), and I agree that it definitely helps with understanding as well as debugging; you can get to the point of being able to decode, slowly, by just staring at a hexdump and "reading" the bits and bytes as if it was a new language.
netheril96 4y ago
My biggest gripe with zstd is that it is difficult to find a Windows program to operate on them. I had to manually download zstd and a bunch of dependencies manually from msys2 in order to decompress.
- wolf550e 4y ago
  The official cli windows binaries are here: https://github.com/facebook/zstd/releases/ https://github.com/facebook/zstd/releases/ Afaik, they don't need any dependencies.
kazinator 4y ago
Is there some smaller version of this, suitable for embedding into programs? $ size /usr/lib/i386-linux-gnu/libzstd.so.1.3.3 text data bss dec hex filename 508674 492 20 509186 7c502 /usr/lib/i386-linux-gnu/libzstd.so.1.3.3 Yikes; half a meg of code! Say I just want something in my C program for saving certain files using less space, and only care about decompressing only those files. Is there some small implementation of a subset of Zstandard: say no more than four or five source files, and 64K of machine code? If I add 500K to the program, I'm going to have to save 500K by compressing the accompanying files, just to begin breaking even. For comparison, libz may not have the decompression speed or the ratio, but it's more than four times smaller: $ size /lib/i386-linux-gnu/libz.so.1.2.11 text data bss dec hex filename 115537 588 4 116129 1c5a1 /lib/i386-linux-gnu/libz.so.1.2.11 This is more like it! $ size /lib/i386-linux-gnu/libbz2.so.1.0.4 text data bss dec hex filename 60511 3520 4 64035 fa23 /lib/i386-linux-gnu/libbz2.so.1.0.4 Text 60511. Is there some configuration of Zstandard which can cram into comparable code space?
- nigeltao 4y ago
  https://github.com/facebook/zstd/tree/dev/doc/educational_decoder https://github.com/facebook/zstd/tree/dev/doc/educational_de... is a self-contained zstd decoder. I get a 64 KB dynamically linked executable after running "make" in that directory. $ size harness text data bss dec hex filename 24615 704 40 25359 630f harness
- kazinator 4y ago
  bzip2 comes out the clear winner in compressing TXR Lisp compiled files, and has the smallest code size (by far) in its shared library form: 0:[0519:020528]:sun-go:~/txr/stdlib$ ls -l compiler.tlo* -rw-rw-r-- 1 kaz kaz 211119 May 12 11:06 compiler.tlo -rw-rw-r-- 1 kaz kaz 37615 May 19 02:01 compiler.tlo.bz2 -rw-rw-r-- 1 kaz kaz 49200 May 19 02:01 compiler.tlo.gz -rw-rw-r-- 1 kaz kaz 38360 May 19 02:04 compiler.tlo.xz -rw-rw-r-- 1 kaz kaz 52387 May 19 02:02 compiler.tlo.zst 0:[0519:020535]:sun-go:~/txr/stdlib$ size /lib/i386-linux-gnu/libbz2.so.1.0.4 text data bss dec hex filename 60511 3520 4 64035 fa23 /lib/i386-linux-gnu/libbz2.so.1.0.4 0:[0519:020544]:sun-go:~/txr/stdlib$ size /lib/i386-linux-gnu/libz.so.1.2.11 text data bss dec hex filename 115537 588 4 116129 1c5a1 /lib/i386-linux-gnu/libz.so.1.2.11 0:[0519:020653]:sun-go:~/txr/stdlib$ size /lib/i386-linux-gnu/liblzma.so.5.2.2 text data bss dec hex filename 166862 900 4 167766 28f56 /lib/i386-linux-gnu/liblzma.so.5.2.2 0:[0519:020658]:sun-go:~/txr/stdlib$ size /usr/lib/i386-linux-gnu/libzstd.so.1.3.3 text data bss dec hex filename 508674 492 20 509186 7c502 /usr/lib/i386-linux-gnu/libzstd.so.1.3.3 Code bloat: bz2 < z < xz < zstd Compression: bz2 > xz > z > zstd zstd with level -19 compression beats default gz to third place: -rw-rw-r-- 1 kaz kaz 41163 May 19 02:17 compiler.tlo.zst-19 gzip squeezes another 2K out with level -9, staying third: -rw-rw-r-- 1 kaz kaz 47206 May 19 02:19 compiler.tlo.gz-9 xz's output doesn't change at -9 level (relative to its default -6).

175 points by stargrave 4y ago

This post talksabout literals still being encoded with huffman tables and stuff. However, it appears this must be optional, because short strings compressed with zstd pass through 'unencoded', apart from a header and footer added: $ echo "hello world" | zstd | hexdump -C 00000000 28 b5 2f fd 04 58 61 00 00 68 65 6c 6c 6f 20 77 |(./..Xa..hello w| 00000010 6f 72 6c 64 0a 8c 6d 7d 20 |orld..m} | As an aside, I'm fairly disappointed that zstd didn't use a single byte header for uncompressible strings, so that they could guarantee that the compressed data will never be more than 1 byte larger than the source data. That property is very useful where lots of tiny strings are being compressed, such as in a database.

wolf550e 4y ago
This is an uncompressed block. For database records or log lines, you should use dictionary trained on likely data. Then short records compress well. See what facebook does with zstd (they employ its author): https://engineering.fb.com/2018/12/19/core-data/zstandard/ https://engineering.fb.com/2018/12/19/core-data/zstandard/
pcwalton 4y ago
The first 4 bytes are the magic number and the last 4 bytes are the checksum [1] which you could always just chop off if you wanted (it's legal to omit the checksum, see the spec). That would get the total overhead down to 5 bytes. [1]: https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#zstandard-frames https://github.com/facebook/zstd/blob/dev/doc/zstd_compressi...
terrelln 4y ago
Zstd has a 4-byte magic number, which is used to check if the data is zstd encoded. In addition to that, this example has a 2 byte frame header (including the decompressed size), a 3 byte block header, and a 4 byte checksum at the end (which can be disabled with `--no-check`). Zstd does have a mode that passes-through incompressible data, both for incompressible literals, and completely incompressible blocks (128 KB chunks). For small inputs, we recommend using dictionary compression. But even with dictionary compression, because of our header costs, you won't generally see benefits until your data is at least ~50 bytes. But YMMV depending on your data.

keyle 4y ago

Interesting, I've never heard of it before. I was aware that there were better standards than zip, however adoption was always an issue with licenses. If this is as open, is it purely an issue of wide adoption? We're stuck in the C++ paradigm (I'm sure there is a better name), where everyone agree this isn't "ideal", that there is better, but not widely adopted enough?

learndeeply 4y ago
Your questions are answered in the first sentence of the article. > Zstd or Zstandard (RFC 8478, first released in 2015) is a popular modern compression algorithm. It’s smaller (better compression ratio) and faster than the ubiquitous Zlib/Deflate (RFC 1950 and RFC 1951, first released in 1995).
c0balt 4y ago
The article already mentions zstd but for an IMO simple and accurate overview take a look at the recommendations from https://jolynch.github.io/posts/use_fast_data_algorithms/ https://jolynch.github.io/posts/use_fast_data_algorithms/ .
wolf550e 4y ago
Unless you have to use deflate/zlib/gzip/zip for compatibility or you're very constrained in amount of memory, you should always prefer zstd over deflate - it's always both faster and compresses better, so there is no tradeoff and using deflate is just wasteful. Google's brotli is a close competitor and got supported in Chrome first, so HTTP uses that more than it uses zstd. zstd has a very wide range of compression levels to tradeoff compression ratio for CPU time, much wider than the difference between gzip -1, gzip -6 (default) and gzip -9. zstd -3 is default. zstd -1 is very fast, zstd -19 compresses really well. zstd also has a long range mode, for archives that contain the same file a gigabyte away. gzip's window is only 32KB in size, so it can't notice any repetitions more than that far away. LZ4 by the same author as zstd aims at even faster compression, at the cost of worse compression ratio. It makes sense to use over fast network. Google snappy and LZO are the usual competitors. When you want better compression than zstd (at the cost of CPU time, both during compression and during decompression), use PPMD for natural language text or source code and use LZMA (xz) for other things.

dietrichepp 4y ago

Nice writeup! FSE, one of the parts of Zstandard, is a very elegant entropy coder. If you're following this section of the article, you might be wondering how it's encoded, since the encoding process is somewhat less obvious than for Huffman coding. What causes trouble is simply, "Once you've emitted a symbol, how do you know that the next symbol is in the range [BL, BL+2^NB)?" The answer is that the encoding is performed backwards, starting from the end of the input, and the table is constructed so that for any symbol emitted, you can find a previous symbol so that the current symbol is in the previous symbol's [BL, BL+2^NB) range. There's another writeup about FSE here, which goes into more depth about FSE (rather than talking about Zstandard in general): http://fastcompression.blogspot.com/2013/12/finite-state-entropy-new-breed-of.html http://fastcompression.blogspot.com/2013/12/finite-state-ent... My intuition tells me that there's probably a homomorphism from Huffman coding to FSE that preserves the encoded size exactly, but I haven't done the math to check.

FullyFunctional 4y ago
That smells an awful lot like arithmetic coding. Is it just a reformulation (genuinely curious)?
- dietrichepp 4y ago
  I wouldn't describe it as a reformulation, but as a different solution to the same problem. The goal of entropy coders in general is to encode each symbol using -log P bits, where P is the probability of that symbol. Huffman, arithmetic encoding, range coding, and ANS / FSE are different systems that solve this problem. https://en.wikipedia.org/wiki/Entropy_coding https://en.wikipedia.org/wiki/Entropy_coding Huffman coding works by using an integer number of bits for each symbol and then creating a prefix-free code. This is computationally cheap but wastes some space. Arithmetic coding works using interval arithmetic. Each symbol corresponds to an interval, and you multiply each successive interval together, renormalizing after each step. This is close to optimal but computationally expensive. ANS / FSE work by using a state machine. A particular symbol can be coded using a variable number of bits, but that number is an integer. The average number of bits is close to -log P. Like Huffman coding, coding and decoding is fast--each symbol uses an integer number of bits, and you don't need to do any multiplication. Like Arithmetic coding, it is close to optimal in terms of space used. These techniques share one thing in common--you only need to record symbol frequency in order to recreate the codebook. In Huffman coding, you typically encode the symbol frequency log 2 (i.e. the symbol length). In FSE and arithmetic coding, you typically encode the symbol frequency as a dyadic rational. You use a deterministic process to recreate the exact same codebook given these frequencies. (Then the question is, "How do you encode the symbol frequency efficiency?" For Deflate, the answer is, funny enough, the symbol lengths for the Huffman codes are themselves encoded using Huffman codes! The codebook for that Huffman code is then coded using a fixed codebook.)

terrelln 4y ago

Awesome post, it's great to see someone diving into the details of Zstd! Maintainer of Zstd here, AMA

Rafuino 4y ago
Ohh, don't mind if I do! I'm working on CPU libraries to improve compression performance. Does zstd benefit today from ISAs like AVX-512, AVX-2, etc.? Do you foresee the need in the future to offload compression/decompression to an accelerator? Another Q, what hardware platform(s) do you use to test new versions of zstd for regressions/improvements? I see the Github page shows a consumer CPU in use, but what about server CPUs pushing maximum throughput with many threads/cores?
- terrelln 4y ago
  > Does zstd benefit today from ISAs like AVX-512, AVX-2, etc.? Zstd benefits mostly from BMI(2), it takes advantage of shlx, shrx, and bsf during entropy (de)coding. We do use SSE2 instructions in our lazy match finder to filter matches based on a 1-byte hash, in a similar way that F14 and Swiss hash tables do. We also use vector loads/stores to do our copies during decompression. We don't currently benefit from AVX2. There are strategies to use AVX2 for Huffman & FSE compression, but they don't fit well with zstd's format, so we don't use them there. Additionally, our latest Huffman decoder is very competitive with the AVX2 decoder on real data. And a Huffman format that is fast for AVX2 is often slow for scalar decoding, so it is hard to opt into it unless you know you will only run on CPUs that support AVX2. Lastly, AVX2 entropy decoders rely on a fast gather instruction, which isn't available on AMD machines. > Do you foresee the need in the future to offload compression/decompression to an accelerator? Yes, there are trends in this direction. Intel QAT supports zlib hardware acceleration. The PS5 has accelerated decompression. I'm pretty sure Microsoft has also released a paper about a HW accelerated algorithm. Compression & decompression take up a significant portion of datacenter CPU, so it makes sense to hardware accelerate it. > what hardware platform(s) do you use to test new versions of zstd for regressions/improvements? We mostly test and optimize for x86-64 CPUs, on a mix of server and consumer CPUs. But, we also test on ARM devices to make sure we don't have major regressions. We've found that optimizing for x86-64 has done a good job of getting good baseline performance on other architectures, though there is certainly room left to improve.

userbinator 4y ago

LZ followed by entropy coding seems to be the general strategy of choice for many compression algorithms, so this design of Zstandard was quite familiar to me. The FSE reminded me of the Q/MQ/QM-coder algorithm which is used in some image formats (and is surprisingly simple yet powerful, although relatively slow.) Nonetheless, I like to learn a file format by studying a worked example at the bits and bytes level. I did the same when I was playing around with various image and video codecs (JPEG, GIF, MPEG-1/2, etc.), and I agree that it definitely helps with understanding as well as debugging; you can get to the point of being able to decode, slowly, by just staring at a hexdump and "reading" the bits and bytes as if it was a new language.

netheril96 4y ago

My biggest gripe with zstd is that it is difficult to find a Windows program to operate on them. I had to manually download zstd and a bunch of dependencies manually from msys2 in order to decompress.

wolf550e 4y ago
The official cli windows binaries are here: https://github.com/facebook/zstd/releases/ https://github.com/facebook/zstd/releases/ Afaik, they don't need any dependencies.

kazinator 4y ago

Is there some smaller version of this, suitable for embedding into programs? $ size /usr/lib/i386-linux-gnu/libzstd.so.1.3.3 text data bss dec hex filename 508674 492 20 509186 7c502 /usr/lib/i386-linux-gnu/libzstd.so.1.3.3 Yikes; half a meg of code! Say I just want something in my C program for saving certain files using less space, and only care about decompressing only those files. Is there some small implementation of a subset of Zstandard: say no more than four or five source files, and 64K of machine code? If I add 500K to the program, I'm going to have to save 500K by compressing the accompanying files, just to begin breaking even. For comparison, libz may not have the decompression speed or the ratio, but it's more than four times smaller: $ size /lib/i386-linux-gnu/libz.so.1.2.11 text data bss dec hex filename 115537 588 4 116129 1c5a1 /lib/i386-linux-gnu/libz.so.1.2.11 This is more like it! $ size /lib/i386-linux-gnu/libbz2.so.1.0.4 text data bss dec hex filename 60511 3520 4 64035 fa23 /lib/i386-linux-gnu/libbz2.so.1.0.4 Text 60511. Is there some configuration of Zstandard which can cram into comparable code space?

nigeltao 4y ago
https://github.com/facebook/zstd/tree/dev/doc/educational_decoder https://github.com/facebook/zstd/tree/dev/doc/educational_de... is a self-contained zstd decoder. I get a 64 KB dynamically linked executable after running "make" in that directory. $ size harness text data bss dec hex filename 24615 704 40 25359 630f harness
kazinator 4y ago
bzip2 comes out the clear winner in compressing TXR Lisp compiled files, and has the smallest code size (by far) in its shared library form: 0:[0519:020528]:sun-go:~/txr/stdlib$ ls -l compiler.tlo* -rw-rw-r-- 1 kaz kaz 211119 May 12 11:06 compiler.tlo -rw-rw-r-- 1 kaz kaz 37615 May 19 02:01 compiler.tlo.bz2 -rw-rw-r-- 1 kaz kaz 49200 May 19 02:01 compiler.tlo.gz -rw-rw-r-- 1 kaz kaz 38360 May 19 02:04 compiler.tlo.xz -rw-rw-r-- 1 kaz kaz 52387 May 19 02:02 compiler.tlo.zst 0:[0519:020535]:sun-go:~/txr/stdlib$ size /lib/i386-linux-gnu/libbz2.so.1.0.4 text data bss dec hex filename 60511 3520 4 64035 fa23 /lib/i386-linux-gnu/libbz2.so.1.0.4 0:[0519:020544]:sun-go:~/txr/stdlib$ size /lib/i386-linux-gnu/libz.so.1.2.11 text data bss dec hex filename 115537 588 4 116129 1c5a1 /lib/i386-linux-gnu/libz.so.1.2.11 0:[0519:020653]:sun-go:~/txr/stdlib$ size /lib/i386-linux-gnu/liblzma.so.5.2.2 text data bss dec hex filename 166862 900 4 167766 28f56 /lib/i386-linux-gnu/liblzma.so.5.2.2 0:[0519:020658]:sun-go:~/txr/stdlib$ size /usr/lib/i386-linux-gnu/libzstd.so.1.3.3 text data bss dec hex filename 508674 492 20 509186 7c502 /usr/lib/i386-linux-gnu/libzstd.so.1.3.3 Code bloat: bz2 < z < xz < zstd Compression: bz2 > xz > z > zstd zstd with level -19 compression beats default gz to third place: -rw-rw-r-- 1 kaz kaz 41163 May 19 02:17 compiler.tlo.zst-19 gzip squeezes another 2K out with level -9, staying third: -rw-rw-r-- 1 kaz kaz 47206 May 19 02:19 compiler.tlo.gz-9 xz's output doesn't change at -9 level (relative to its default -6).