4 ms·
Current leader of the Large Text Compression Benchmark is NNCP (compression using neural networks), also by Fabrice Bellard: https://bellard.org/nncp/ https://
by omoikane 9mo ago
Current leader of the Large Text Compression Benchmark is NNCP (compression using neural networks), also by Fabrice Bellard:
https://bellard.org/nncp/ https://bellard.org/nncp/
Also, nncp-2024-06-05.tar.gz is just 1180969 bytes, unlike ts_zip-2024-03-02.tar.gz (159228453 bytes, which is bigger than uncompressed enwiki8).
- smusamashah 9mo agoDoesn't this fit the Hutter Prize conditions that is mentioned in other comment here https://news.ycombinator.com/item?id=46595109 https://news.ycombinator.com/item?id=46595109
- WithinReason 9mo agoIt's too slow for that. The Hutter prize is CPU only so neural network solutions (which are the most interesting IMO) are effectively excluded. You need to generate 11 574 characters per second on the CPU only for decompression, and the compression time also counts and has to be below 24 hours in total.
- cztomsik 9mo agoIIRC all the cmix submissions are using NN (and were for long time)
- WithinReason 9mo agowell yes, small LSTMs I think, which is far from LLM territory
- wiz21c 9mo agowhile impressive, it's a very specific case of compression: english text. It may be in use in many places but there are many more things to compress. It'd be nice to have a comparison here: https://morotti.github.io/lzbench-web/?dataset=silesia/sao&machine=desktop https://morotti.github.io/lzbench-web/?dataset=silesia/sao&m...