4 ms·
Obviously now that people have seen that this is possible it will be replicated in short order whether they release it or not.
by yogthos 7y ago
Obviously now that people have seen that this is possible it will be replicated in short order whether they release it or not.
- pigscantfly 7y agoFor those interested, this open-source replication of the dataset is a good place to start https://skylion007.github.io/OpenWebTextCorpus/ https://skylion007.github.io/OpenWebTextCorpus/ - obviously, you'll need access to some large GPUs as well. I'm currently training one across four cards, although I'd like to throw more power at it. Very interested in altering the architecture to reduce memory requirements (without unduly effecting the loss). Excited to see what the community comes up with.
- petters 7y agoYes, but training it is very expensive.
- yogthos 7y agoBut that only needs to be done once. Also, it turns out that the current crop of neural nets is incredibly inefficient https://www.technologyreview.com/s/613514/a-new-way-to-build-tiny-neural-networks-could-create-powerful-ai-on-your-phone/ https://www.technologyreview.com/s/613514/a-new-way-to-build...