A similar project by Bellard is https://bellard.org/nncp/ https://bellard.org/nncp/ It's a transformer that trains itself on the user's input and nothing else.

A similar project by Bellard is https://bellard.org/nncp/ https://bellard.org/nncp/ It's a transformer that trains itself on the user's input and nothing else. The problem is that for the first N megabytes of input the compression ratio is going to be really bad because the model is only just starting to figure things out. Once you get a couple GB of training data things get much better. That's why in practice grabbing a pre-trained model is more practical.