Sure! The current best achieved compression for English text (well, specifically, Wikipedia articles) is Fabrice Bellard's nncp, which uses... transformers. htt

Sure! The current best achieved compression for English text (well, specifically, Wikipedia articles) is Fabrice Bellard's nncp, which uses... transformers. http://mattmahoney.net/dc/text.html#1085 http://mattmahoney.net/dc/text.html#1085 Unfortunately I don't think anyone has posted any trained models after running any inputs through it, which could be used to repeat the experiment. I would try it, but I don't have a AVX2 CPU. However, most research into reducing the model perplexity of text (increasing the compression ratio when the model is used for encoding) uses fixed language models which are learnt offline. Obviously, they use transformers too. You can see some of them here: https://paperswithcode.com/sota/language-modelling-on-wikitext-103 https://paperswithcode.com/sota/language-modelling-on-wikite... However (token) perplexity is only comparable across models if the same tokenizer is used. The above page seems to compare papers using the GPT-2 tokenizer. I would assume that if you controlled for differences in tokenizers by just measuring bits-per-word/byte, GPT-4 without RLHF would achieve the highest compression. It should be obvious that RLHF increases perplexity because it changes the training objective to something other than predicting text.