3 ms·
We solved the proper dataset part at least. https://arxiv.org/abs/2101.00027 https://arxiv.org/abs/2101.00027 My contribution was around 19,000 books.
by sillysaurusx 4y ago
We solved the proper dataset part at least. https://arxiv.org/abs/2101.00027 https://arxiv.org/abs/2101.00027
My contribution was around 19,000 books.
- hooande 4y agoisn't Common Crawl much, much larger than this? ~6 pebibytes from what I remember
- sillysaurusx 4y agoYeah, but the hard part is filtering. It’s pretty easy to scrape a massive amount. Turning it into quality training data is the trick. bmk was the magician there. https://twitter.com/nabla_theta?s=21&t=Gt6YrATJHnmY046MdzhYDg https://twitter.com/nabla_theta?s=21&t=Gt6YrATJHnmY046MdzhYD...
- swyx 4y ago> My contribution was around 19,000 books what does this mean? not meaning to cross examine you, just curious how people contribute to The Pile since it seemingly appeared out of nowhere
- sillysaurusx 4y agoNot at all, I love talking about it. I was convinced that a model needed to be able to read like we do. And what do we do when we read? Pick up a book. That turns out to be surprisingly hard, at least for training data. Step one is to acquire the books. Step two is to turn them into a readable format for computers. Both steps were very hard. I lucked out on step one because The Eye happened to host all of bibliotok, which came to around 30k books or so. Trouble is, lots of those are PDFs. And although humans are great at reading those, they fucking suck for blind people. And a gpt is a blind person in a sense, because it needs to follow a linear sequence of words — something that PDFs are horrible at giving. But one day I realized that epubs were merely html files, and aaronsw happened to write an amazing html to text converter. I had to hack it to fix a few corner cases. But after a few days, I ran it across all 19,000 epubs I spidered, then zipped the whole thing up and called it books3: https://twitter.com/theshawwn/status/1320282149329784833?s=46&t=mecILIS2Eh73ASUY2N95YQ https://twitter.com/theshawwn/status/1320282149329784833?s=4... It’s one of the larger components of the pile, I think around 35%. Which is quite the hefty sum when it’s purely text. I still have a hard time wrapping my head around just how mindbogglingly big 800GB of text is.
- swyx 4y agooh wow, that was a surprisingly awesome story. thanks for sharing! this maaaay be covered in the Pile's writeup (which i have not yet read) but i wonder who was curating the overall "mix" of the content. seems easily biased to, say, public domain books, since the corpus is easily available. when people say things like "GPT3 has been trained on all of the internet" i suspect this is a gross exaggeration. In reality it's just C4/commoncrawl, so that's like 800GB of text.
- sillysaurusx 4y agoStella! She’s awesome. https://twitter.com/blancheminerva?s=21&t=Gt6YrATJHnmY046MdzhYDg https://twitter.com/blancheminerva?s=21&t=Gt6YrATJHnmY046Mdz... Also bmk. https://twitter.com/nabla_theta?s=21&t=Gt6YrATJHnmY046MdzhYDg https://twitter.com/nabla_theta?s=21&t=Gt6YrATJHnmY046MdzhYD... They did the legwork of writing the paper and getting everything into a presentable format. A bunch of other people helped too; I wasn’t as involved as I could’ve been. It was all discord-based. As far as I know it was the first serious research collaboration to happen solely via chatroom. bmk also got the 50GB of code from GitHub, I think. So that’s where GPT-J’s coding ability likely came from.
- swyx 4y agothank you for those names, ive added to my watchlist. could be a good story to write up, i think the people who do the hard work behind all the datasets dont get any of the glory of the ML models that get built on top of it