15 ms·
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- leogao 6y agoTwitter thread: https://twitter.com/nabla_theta/status/1345130412532645888 https://twitter.com/nabla_theta/status/1345130412532645888
- bratao 6y agoI´m super excited by this dataset. The EleuterAI team is stellar and many great things are coming soon from them!
- legatus 6y agoI think it's worth noting that EleutherAI is a grassroots collection of researchers, which distinguishes it from academia/industry labs. As part of their work on democratizing AI, they're now hoping to replicate GPT-3 and release it for free (unlike OpenAI's API). I would encourage everyone interested to join their discord server (https://discord.gg/BK2v3EJ https://discord.gg/BK2v3EJ) -- they're extremely friendly and I think it's a project worth contributing to.
- joe_the_user 6y agoAny idea how large dataset used to train GPT-3 was?
- arugulum 6y ago570GB of Common Crawl post-filtering, but only 40% of CC data was seen even once during training, though CC is only 60% of the training data. You could work through the math to find the rough size of GPT-3's training data, but it sounds like The Pile is of comparable size.
- leogao 6y agoYeah, the Pile is approximately the size of the GPT-3 training data, which is not a coincidence--one major reason we created the Pile (though certainly not the only one) was for our GPT-3 replication project.
- odnes 6y agoHow are they sourcing/funding the compute to train these massive models?
- luto 6y agothey'll probably run it on scientific clusters of various universities, or on collections of idle lab desktop machines. Both of these tend to sit idle a lot of the time, based on my experience at uni in Europe.
- Alvion_Bleeds 6y agoConnor Leahy, who I think is a sort of BDFL figure for ElutherAI, mentioned in a Slatestarcodex online meetup I attended that Google donated millions of dollars worth of preemptable TPU credits to the project. There is a video of the meetup on YouTube somewhere. Struck me as a really smart kid with a lot of passion.
- chillee 6y agoHaha Connor (although one of the main participants) definitely isn't a BDFL - we don't have any BDFLs :) We don't really have much of a hierarchy at all - it's mostly just a collection of researchers of widely varying backgrounds all interested in ML research.
- stellaathena 6y agoI'm not sure what a BDFL figure is, but Google does not give us millions of dollars. We are a part of TFRC, a program where researchers and non-profits can borrow TPUs when they're not being used. You could say that we are indirectly funded as a result, but it's nowhere near millions of dollars and it doesn't reflect any kind of special relationship with Google.
- o-__-o 6y agoBenevolent dictator for life
- leogao 6y ago
- pizza 6y agoRelated: is it possible for me to download/try out a pretrained GPT-Neo?
- deleted 6y ago[deleted]
- stellaathena 6y agoNot currently. The code is on GitHub, but we do not have a public-facing model. We felt that the world doesn't need another GPT-2 replica, but if there's interest we can look into doing so though. We are planning on making our GPT-3 replica public facing though. https://github.com/EleutherAI/gpt-neo https://github.com/EleutherAI/gpt-neo
- thereticent 6y agoLove the name. It's what we called our shared FTP-served collection of mp3s during college 15-19 years ago. Yours is a MUCH more impressive amount of information!
- eutectic 6y agoI'm kind of surprised there's 800GB of text in the world.
- luto 6y agoheh, yeah! for comparison: the English Wikipedia is around 40 GB of text. https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Size_of_the_English_Wikipedia_database https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Si...
- pbourke 6y agoIt’s not that much, actually. The average English word is 4.7 letters long. Let’s round up to 5 and add 1 for a space to make 6 characters. Novels are around 90,000 words long. So 800G of pure text represents 800G / 6 char per word / 90,000 words per book =~ 1.5M books. The Library of Congress has 39M books, not to mention all the text produced that’s exclusively online.
- d33 6y agoThat makes me wonder what's the upper bound of storage required to contain all of the text humans had ever written. 1k times as much? 10k? Either way, I have a feeling that all of the drives would probably fit in a regular apartment room. Crazy how we went from first computers spanning across entire floors to fitting all of human thoughts in such tight space.
- Aerroon 6y agoThe entire Library of Congress according to the earlier calculations would fit into 21 TB. There's already a 100 TB SSD that's the size of a 3.5" HDD.[0] 10000 times the Library of Congress would fit onto 2100 of these drives. The dimensions for this 3.5" drive are 26.1 x 147 x 101.8 mm.[1] That's a volume of roughly 391 cm^3. 2100 of them take up 821,100 cm^3. A Fractal Design Define 7 has the dimensions of 547 x 240 x 475 mm. That's a volume of 62,358 cm^3.[2] You could fit 2100 of those SSDs into the same volume as about 13.2 Fractal Design Define 7 cases. That's roughly the size of a bookshelf. You could fit the books of 10000 Libraries of Congress onto one bookshelf. Now imagine using tape for this. Edit: I'm actually not sure whether tape has better density. [0] https://www.techradar.com/news/at-100tb-the-worlds-biggest-ssd-gets-an-eye-watering-price-tag https://www.techradar.com/news/at-100tb-the-worlds-biggest-s... [1] https://www.newegg.com/nimbus-data-dc-100tb/p/2U3-002M-00004 https://www.newegg.com/nimbus-data-dc-100tb/p/2U3-002M-00004 [2] https://www.tomshardware.com/reviews/fractal-design-define-7 https://www.tomshardware.com/reviews/fractal-design-define-7
- 28095787 6y agoHi pretty! Are you seeking a wealthy daddy for financial support and online base relationship with $1000 weekly allowance and $300 for first person to add me... Whatsapp:+14752340571 Email: Michealwebb002@gmail.com
- sillysaurusx 6y agoI'm sad they dropped Literotica from the dataset. It's available in the old "preliminary components" archive, if anyone wants to train a Literotica-Enron-Email GPT: https://the-eye.eu/public/AI/pile_preliminary_components/ https://the-eye.eu/public/AI/pile_preliminary_components/ I think The Pile is probably one of the most important AI projects of the last year or so, at least for lone wolf researchers like me. Gathering training data at scale can be excruciatingly difficult. (Perhaps someone will do something similar for GAN training one day: a large dataset for image modeling would help a lot.) By the way, consider contributing to The Eye: https://the-eye.eu/ https://the-eye.eu/ Without them, I’m not sure any of us would have been able to host the datasets we gathered — or organized torrent seeds, or fielded DMCA complaints, etc. So I feel The Eye deserves an explicit shoutout for being such an asset to the AI community, along with TFRC and Eleuther.
- 6gvONxR4sf7o 6y agoI feel the opposite way about the eye. The data hosted by the eye isn’t theirs to give away. Their approach feels disingenuous.
- sillysaurusx 6y agoI struggled with similar feelings. But, what convinced me, is that they take great care to remain compliant with DMCA. Pirates don’t. The difference with The Eye is that they thoroughly vet the legitimacy of each DMCA claim. The claimant is required to show proof that they are the legal copyright holder. And The Eye seems willing to call bluffs: on more than one occasion they have dealt with bogus DMCAs in ways where normal companies would simply give up. Ultimately, their actions are legal. And for us in the AI community, it was something like the hand of God reaching down to bless us with a guardian angel. The reproducibility situation is getting worse each week, and much of that is due to the fact that realistic datasets can’t be distributed without fear of reprisals. That said, I respect your feelings on the matter too. I think it’s equally valid to feel uncomfortable. I just take solace in the fact that it’s legal.
- 6y ago
- w1nk 6y agoWill this dataset be able to produce a model that's actually better than GPT-3? One of the things mentioned in the paper is that the dataset had some buggy filtering applied to it (https://arxiv.org/pdf/2005.14165.pdf https://arxiv.org/pdf/2005.14165.pdf, page 9), that minimally impacted benchmarking some, there's a whole section on how they tried to deal with it. The gist though is that a cleaner training run, even on slightly more data, may (should?) actually produce something a bit better. Does anyone know if OpenAI has retrained/updated gpt-3 yet?
- forgingahead 6y agoThis is a great effort, and it's important to have datasets like this available to democratise ML learning and work. One small comment: It would be great for this (and other) datasets to give a quick "sample data" file - preferably one that doesn't need to be downloaded to be viewed. Even a screenshot of some of the data would be useful for people browsing to get a quick understanding of the actual content, and how it is formatted. Downloading gigabytes of data just to have a look isn't practical.
- remexre 6y agoThe test and validation datasets are under half a gig; I agree it'd be nice if there were a few example items and perhaps a JSON Schema [0] for it, though. I'll reproduce a line here: {"text": "Roman Catholic Diocese of Tambacounda\n\nThe Roman Catholic Diocese of Tambacounda () is a diocese located in the city of Tambacounda in the Ecclesiastical province of Dakar in Senegal.\n\nHistory\n August 13, 1970: Established as Apostolic Prefecture of Tambacounda from the Diocese of Kaolack and Diocese of Saint-Louis du S\u00e9n\u00e9gal\n April 17, 1989: Promoted as Diocese of Tambacounda\n\nSpecial churches\n The cathedral is Cath\u00e9drale Marie Reine de l\u2019Univers in Tambacounda, which is located in the Medina Coura neighborhood of the town.\n\nLeadership\n Bishops of Tambacounda (Roman rite)\n Bishop Jean-No\u00ebl Diouf (since 1989.04.17)\n Prefects Apostolic of Tambacounda (Roman rite) \n Fr. Cl\u00e9ment Cailleau, C.S.Sp. (1970.08.13 \u2013 1986.04.24)\n\nSee also\nRoman Catholicism in Senegal\n\nReferences\n\nExternal links\n GCatholic.org\n Catholic Hierarchy \n\nCategory:Roman Catholic dioceses in Senegal\nCategory:Tambacounda\nCategory:Christian organizations established in 1970\nCategory:Roman Catholic dioceses and prelatures established in the 20th century", "meta": {"pile_set_name": "Wikipedia (en)"}} [0]: https://json-schema.org/ https://json-schema.org/
- gwern 6y agoI believe the paper includes random samples at the end.
- mrconter1 6y agoIt includes two example snippets from each dataset.
- floatingatoll 6y agoTo clarify, they mean diverse as in “unconnected datasets”.
- durnygbur 6y agoSomeone explain me how this will not stir the wrath of copyright echelons, especially American ones? Or once I get PhD I can torrent movies again?
- unixhero 6y agoHow can I start working on this dataset? Are there any Python GPT libraries?
- mrfusion 6y agoWhy not throw Wikipedia into this collection?
- ValueNull 6y agoHopefully not the Scots version
- stellaathena 6y ago... is it in the collection?