3 ms·
The Facebook Llama paper[1] calls out a number of the ones they used. Briefly, they include: - CommonCrawl - Wikipedia - Github - StackExchange - arXiv a
by mindcrime 4y ago
The Facebook Llama paper[1] calls out a number of the ones they used.
Briefly, they include:
- CommonCrawl
- Wikipedia
- Github
- StackExchange
- arXiv
and "Books" which appears to be made up from Project Gutenberg books among others.
[1]: https://scontent-iad3-1.xx.fbcdn.net/v/t39.8562-6/333078981_693988129081760_4712707815225756708_n.pdf?_nc_cat=108&ccb=1-7&_nc_sid=ad8a9d&_nc_ohc=fskqjIsP1vwAX-_NNhB&_nc_ht=scontent-iad3-1.xx&oh=00_AfBUSnw_go9eRLVAfuVJ3BoItoxnsfO2qbFEC_FofFq1gA&oe=64216DE2 https://scontent-iad3-1.xx.fbcdn.net/v/t39.8562-6/333078981_...