See a related benchmark: http://www.mattmahoney.net/dc/text.html http://www.mattmahoney.net/dc/text.html In particular, Fabrice Bellard is leading with a trans

See a related benchmark: http://www.mattmahoney.net/dc/text.html http://www.mattmahoney.net/dc/text.html In particular, Fabrice Bellard is leading with a transformer model for “enwik9” that which is 10 times larger than the one used in Hutter. It’s doing quite well with enwik8 too, but perhaps the issue is that the “economy of scale” with on the fly training hasn’t caught up in the smaller benchmark yet. Transformers are definitely lossy, but afaict Bellard’s entry uses the probabilities generated by the model to create an encoding that used fewer bits