4 ms·
IDK why, but BZIP2 seems to do somewhat better that other compression algorithms for natural language text: $ curl https://www.gutenberg.org/cache/epub/11/
by Intralexical 2y ago
IDK why, but BZIP2 seems to do somewhat better that other compression algorithms for natural language text:
$ curl https://www.gutenberg.org/cache/epub/11/pg11.txt | bzip2 --best | wc
246 1183 48925
Also, ZSTD goes all the way up to `--ultra -22` plus `--long=31` (4GB window— Irrelevant here since the file fits in the default 8MB anyway).
- pastage 2y agoYou can use a preshared dictionary with bzip2 and zstd so you can get that down alot by using different dictionaries depending on certain rules. I dont know if it helps with literature but I had great success in sending databases with free text like that. In the end it was easier to just use one dictionary for everything and just skip the rules.