4 ms·
Repeating this experiment with a more optimal compression algorithm might be interesting too.
by userbinator 3y ago
Repeating this experiment with a more optimal compression algorithm might be interesting too.
- canjobear 3y agoIt doesn't look qualitatively any better with bzip2 or lzma.
- nerdponx 3y agoIn some sense, the feature space learned by a NN is a compression algorithm.
- versteegen 3y agoSure! The current best achieved compression for English text (well, specifically, Wikipedia articles) is Fabrice Bellard's nncp, which uses... transformers. http://mattmahoney.net/dc/text.html#1085 http://mattmahoney.net/dc/text.html#1085 Unfortunately I don't think anyone has posted any trained models after running any inputs through it, which could be used to repeat the experiment. I would try it, but I don't have a AVX2 CPU. However, most research into reducing the model perplexity of text (increasing the compression ratio when the model is used for encoding) uses fixed language models which are learnt offline. Obviously, they use transformers too. You can see some of them here: https://paperswithcode.com/sota/language-modelling-on-wikitext-103 https://paperswithcode.com/sota/language-modelling-on-wikite... However (token) perplexity is only comparable across models if the same tokenizer is used. The above page seems to compare papers using the GPT-2 tokenizer. I would assume that if you controlled for differences in tokenizers by just measuring bits-per-word/byte, GPT-4 without RLHF would achieve the highest compression. It should be obvious that RLHF increases perplexity because it changes the training objective to something other than predicting text.
- vintermann 3y agoIf I understand it correctly, this implementation re-compresses the prefix 256 times, once for each possible byte continuation. It should be possible to save a lot of work by saving the state of the compressor just before the predicted byte, and reusing it. Also, since gzip works on a limited sliding context window, it's probably pointless to give it a prefix much longer than the window... unless there's something about the algorithm that I'm missing. Same with bzip2 and its block size.
- canjobear 3y agoI'd be happy to work on improving this with someone who understands the internals of the compressors.