I've noticed in the past (trying to optimize traffic for a game with delta compression, variable-length integers, ...) that it's really hard to beat good compre

I've noticed in the past (trying to optimize traffic for a game with delta compression, variable-length integers, ...) that it's really hard to beat good compression with manual tricks. Original word list: https://raw.githubusercontent.com/arpruss/gb-fiver/main/compress/full.txt https://raw.githubusercontent.com/arpruss/gb-fiver/main/comp... $ <full.txt wc # "word count" #lines, words, bytes 12973 12972 77833 $ calc 77833-12973 # bytes minus newlines 64860 # matches the article, to confirm I got the right input data $ <full.txt xz -9 | wc -c 15412 2000 bytes smaller without any optimizations (this even includes the newlines), though now I guess the question is what the size of the xz decompressor is since a few KB actually matter (not like on a regular computer). For some reason bzip2 gets it only to 36K, even worse than gzip (32K) and zstd (29K). Update: Counter-intuitively, stripping the newlines (... | tr -d \\n | ...) results in a higher compressed size with xz. It's surprisingly hard to find a minimal xz decompressor, it doesn't seem as though anyone bothers with this stuff. The Hutter Prize includes the decompressor so that was my first stop, but none of the contestants submitted that (I assume they predate xz). Update#2: Found at least one measurement of xz decompressor at 36K, but it seems to me like this is the general-purpose utility and includes the compressor, help output, etc. http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/text.html