For those interested, this open-source replication of the dataset is a good place to start https://skylion007.github.io/OpenWebTextCorpus/ https://skylion007.gi

For those interested, this open-source replication of the dataset is a good place to start https://skylion007.github.io/OpenWebTextCorpus/ https://skylion007.github.io/OpenWebTextCorpus/ - obviously, you'll need access to some large GPUs as well. I'm currently training one across four cards, although I'd like to throw more power at it. Very interested in altering the architecture to reduce memory requirements (without unduly effecting the loss). Excited to see what the community comes up with.