3 ms·
Huh, They used the pile - that's a pretty interesting choice for a corporate research team?
by buildbot 3y ago
Huh, They used the pile - that's a pretty interesting choice for a corporate research team?
- blackeyeblitzar 3y agoThere are many variants of the pile at this point, including ones with copyrighted content removed. They are probably using one of those, like: https://huggingface.co/datasets/monology/pile-uncopyrighted https://huggingface.co/datasets/monology/pile-uncopyrighted
- buildbot 3y agoOh that’s really great to know about, thank you!
- yunohn 2y agoI don’t see any source for that from their paper. It just says a duplicate version of Pile with a citation linking to the original Pile paper itself. [15] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, An-ish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.