4 ms·
oh wow, that was a surprisingly awesome story. thanks for sharing! this maaaay be covered in the Pile's writeup (which i have not yet read) but i wonder who wa
by swyx 4y ago
oh wow, that was a surprisingly awesome story. thanks for sharing!
this maaaay be covered in the Pile's writeup (which i have not yet read) but i wonder who was curating the overall "mix" of the content. seems easily biased to, say, public domain books, since the corpus is easily available.
when people say things like "GPT3 has been trained on all of the internet" i suspect this is a gross exaggeration. In reality it's just C4/commoncrawl, so that's like 800GB of text.
- sillysaurusx 4y agoStella! She’s awesome. https://twitter.com/blancheminerva?s=21&t=Gt6YrATJHnmY046MdzhYDg https://twitter.com/blancheminerva?s=21&t=Gt6YrATJHnmY046Mdz... Also bmk. https://twitter.com/nabla_theta?s=21&t=Gt6YrATJHnmY046MdzhYDg https://twitter.com/nabla_theta?s=21&t=Gt6YrATJHnmY046MdzhYD... They did the legwork of writing the paper and getting everything into a presentable format. A bunch of other people helped too; I wasn’t as involved as I could’ve been. It was all discord-based. As far as I know it was the first serious research collaboration to happen solely via chatroom. bmk also got the 50GB of code from GitHub, I think. So that’s where GPT-J’s coding ability likely came from.
- swyx 4y agothank you for those names, ive added to my watchlist. could be a good story to write up, i think the people who do the hard work behind all the datasets dont get any of the glory of the ML models that get built on top of it