3 ms·
NNCP, a state-of-the-art lossless text compressor based on a transformer neural network (i.e. containing similar structures to a LLM), can reach a compression r
by unnah 3y ago
NNCP, a state-of-the-art lossless text compressor based on a transformer neural network (i.e. containing similar structures to a LLM), can reach a compression ratio of 10.8% (0.867 bits per byte) on Wikipedia text: https://bellard.org/nncp/ https://bellard.org/nncp/
Compared to NNCP, a hypothetical LLM that "knows" every fact in the English Wikipedia, would not need to keep a lossless record of the entire text of Wikipedia, since many Wikipedia pages repeat facts that also occur on other pages, and instead of reproducing every word it would be enough to paraphrase the ideas. Thus the size of such a LLM could be much smaller than 10% of the size of Wikipedia. How much, it's hard to say...
- moffkalast 3y agoYeah neural nets are amazing compression algorithms since the best way to get the gist of something is to actually understand it, then algorithmically generating the data. Like a calculator having a near infinite compression ratio because it can do math instead of keeping a table of all possible inputs and outputs. Same goes for human brains I suppose, we can only remember as much as we do because of the effective compression our understanding allows for.