10 ms·
The Pile is a 825 GiB diverse, open-source language modelling data set (2020)
- Fornax96 3y agoIt seems the Pile is currently inaccessible. I would be willing to mirror it on Pixeldrain if I can get my hands on it.
- h0p3 3y agov1: magnet:?xt=urn:btih:0d366035664fdf51cfbe9f733953ba325776e667&dn=EleutherAI_ThePile_v1&tr=http%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2F9.rarbg.me%3A2710%2Fannounce&tr=udp%3A%2F%2F3rt.tace.ru%3A60889%2Fannounce&tr=http%3A%2F%2F5rt.tace.ru%3A60889%2Fannounce&tr=udp%3A%2F%2Ftracker.cyberia.is%3A6969%2Fannounce&tr=udp%3A%2F%2Fexodus.desync.com%3A6969%2Fannounce&tr=http%3A%2F%2Fexplodie.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker3.itzmx.com%3A6961%2Fannounce&tr=http%3A%2F%2Ftracker1.itzmx.com%3A8080%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.ch%3A1337%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Fwww.torrent.eu.org%3A451%2Fannounce&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451%2Fannounce&tr=udp%3A%2F%2Fretracker.lanta-net.ru%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.ds.is%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker4.itzmx.com%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.moeking.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.zerobytes.xyz%3A1337%2Fannounce
- turnsout 3y agoThe Pile is pretty old—is this an updated version?
- bt1a 3y agoIt is not. In related news, v2 of the "stack" dataset was recently released > 3.28B unique files belonging to 104.2M github repositories were collected by traversing the Software Heritage 2023-09-06 graph dataset. Additional repository-level metadata was collected from GitHub Archive data up to 2023-09-14. The total uncompressed size of all files is 67.53TB. Near-deduplication was implemented in the pre-processing pipeline on top of exact deduplication. V1 vs V2 by Deduped Size Tokens V1: 2.9TB and 200B V2: 32.1TB and 900B I imagine we'll see some fairly powerful open coding models soon. The ones I'm looking at testing are: dolphincoder-starcoder2-15b-iMat.GGUF CodeFuse-DeepSeek-33B-iMat.GGUF OpenCodeInterpreter-DS-33B-iMat.GGUF starcoder2-15b-instruct-iMat.GGUF more info dataset https://huggingface.co/datasets/bigcode/the-stack-v2 https://huggingface.co/datasets/bigcode/the-stack-v2 gguf quants https://huggingface.co/dranger003 https://huggingface.co/dranger003
- bick_nyers 3y agoDo you happen to know what the v2 dedup size is when compressed? 32.1TB is quite a bit, but if that compresses down to say 3-6TB, it would be much more manageable. Code has a lot of whitespace, repetition, and structure/predictability, so I imagine it would compress better than average text.
- spindump8930 3y agoThose sizes refer to the data before processing and filtering. The actual training size was about 3 TB: The Stack v2 is ten times larger than its predecessor, yielding a raw dataset of 67.5 TB. Through extensive cleaning, filtering, and subsampling of the source code, along with the incorporation of other high-quality code-related datasets, we created a training set of approximately 3TB (900B+ tokens). Source: the paper, Section 10 (https://arxiv.org/pdf/2402.19173.pdf https://arxiv.org/pdf/2402.19173.pdf)
- zellyn 3y agoIs the "books3" dataset mentioned in the Pile paper the one that authors are suing over? The one that includes a whole bunch of popular and copyrighted material?
- Balladeer 3y agoI believe it is. See https://www.wired.com/story/battle-over-books3/ https://www.wired.com/story/battle-over-books3/
- mistrial9 3y agothis list [0] seems like a starting place to look into various legal actions.. not sure how often it is updated e.g. Silverman et al [0] https://originality.ai/blog/openai-chatgpt-lawsuit-list https://originality.ai/blog/openai-chatgpt-lawsuit-list
- bt1a 3y agoPouring one out for the future litigators, jurors, and judges who will have to pore over this inextricable web of legal and technical details
- PeterStuer 3y agoThey'll just let their ai do it over lunch.
- taylorfinley 3y agoYes, from the linked paper: "Books3 is a dataset of books derived from a copy of the contents of the Bibliotik private tracker made available by Shawn Presser (Presser, 2020). Bibliotik consists of a mix of fiction and nonfiction books and is almost an order of magnitude larger than our next largest book dataset (BookCorpus2). We included Bibliotik because books are invaluable for long-range context modeling research and coherent storytelling"
- pimlottc 3y ago
- racee 3y agoi love the Illuminati vibes of The Eye
- brokensegue 3y ago"open source" as in gratis but not as in libre?
- Legend2440 3y agomore like, it's a scrape of the entire internet, use it at your own risk
- fddrdplktrew 3y ago[flagged]
- o11c 3y agoIt should be understood in contrast to most traditional corpora, which are heavily paywalled/restricted ... or else based solely on century-old books. It has long been a major obstacle for linguistics tooling. If the current push of AI companies to get their way (to allow copyright laundering) succeeds, this would almost count as open source by the real definition. If not ... lots of people/companies are committing copyright crimes, some are committing civil infractions, and some may be able to claim fair use.
- jwitthuhn 3y agoIs this still available somewhere? I attempted to download it several months ago and saw the download link 404ing, seems it is still like that.
- TrueDuality 3y agoMost of the distribution for this is via torrents/magnet links and in person hard drive exchanges. I'd go look at some public trackers if you want a copy and don't know someone that already has it. Do be aware that it does include copyrighted content so distribution is piracy.
- Der_Einzige 3y agoAlmost all LLM training datasets include copyrighted content so almost all open source LLM distribution is piracy and almost all API based LLMs, including ChatGPT, are also piracy and copyright laundering. Also, most image-text dataset pairs contain far worse than that. You might want to check out LAION-5B and what stanford researchers have found in there. Technically, anyone who even touched that could in theory be in some serious, serious trouble. I find it quite remarkable that nothing has happened yet.
- littlestymaar 3y agoIt's only piracy if it's private individual doing it, otherwise it's just “ask for forgiveness not for permission”-type Capitalism.
- gosub100 3y agoIt'll be some epic lawsuit like google-v-samsung that will get drawn out for a decade, awarded, and reduced, appealed, etc. where the only winners will be both party's lawyers.
- littlestymaar 3y agoIt's gonna be way worse than this: - OpenAI and others will just settle with MPAA, RIAA and the likes for a revenue stream (a single digit billion a year, likely) + some kind of control over what people can and cannot do with the AI + the access to the technology to produce their own content. - artists will see peanuts from the deal, and the big names are going to be able to stop doing any kind of business with artists which are just expenses in their eyes. They will have been replaced by machines that where trained using their art with no compensation whatsoever. IP is already predatory capitalism, AI will definitely be weaponized against the workers by the owners of the means of “production”.
- DiggyJohnson 3y agoAwesome name. Reminds me of the "original" "Pile" from the Manhattan Project. I read about it in "The Making of the Atomic Bomb" (1986), but presumably it's featured in the recent movie.
- groby_b 3y agoNot really. There's an ultra-brief scene where it's mentioned, but that's it, IIRC. The movie... is a bunch of anecdotes strung together to make a ham-handed point at the end. It was a decent movie if you treat it as a fictional story instead of an actual retelling. I'd stick with the book. (And if you specifically care about Fermi, I recommend "The Last Man Who Knew Everything" by David Schwartz)
- Der_Einzige 3y agoI came so close to getting my Debate document dataset "DebateSum"[1] included into this[2] and I am very sad that it wasn't included to this day: [1] https://github.com/Hellisotherpeople/DebateSum https://github.com/Hellisotherpeople/DebateSum [2] https://github.com/EleutherAI/the-pile/issues/56 https://github.com/EleutherAI/the-pile/issues/56
- joshuakogut 3y ago> If you’d like to contribute it, feel free to submit a PR Stella was waiting for you to submit your dataset. Did you? She closed the ticket many months later.
- Der_Einzige 3y agoThey did a significant amount of work themselves of taking other peoples datasets and including them without the work of the original author needing to submit the full PR to do it. I was then and to this day remain extremely busy Also this was before most datasets were hosted conveniently on huggingface. It's all tears in the rain now.
- joering2 3y agoThe pile can be downloaded here. 404 Not Found nginx 825 GB is a great candidate for torrent use, whatever was under that broken link better be a torrent magnet.
- swatcoder 3y agoWhere do I find the license reproductions and credits/attributions for the content being distributed in this data set? Is it all in there? Are all inclusions compliant? Can I know? I'm open to the argument that generators built with models that consumed copyrighted data may evade copyright obligations on their output, but surely the data sets themselves are bound by any copyright on their content?
- __loam 3y agoThey stole it because they think building their toys is more important than everyone else's rights to the product of their own labor.
- johndough 3y agoI doubt that anyone is going to download and search through over 800 TB just to find a badly formatted copy of some book that could be found much quicker on different websites with better formatting. Authors are losing fractional cents here at most.
- gosub100 3y agoso just like Office Space? (paraphrasing) "We steal a fraction of a cent from each transaction, who do we hurt? Nobody. We just put the remainder into our account!" Sorry that's not how damages are calculated in the US tort system.
- johndough 3y agoI do not know how damages are calculated in the US tort system. What do they say about the books3 dataset? I also think that the case is different here, since in your example, there is a specific amount of money being stolen, while in the books3 case, there is an unspecified amount of money not being made by the authors.
- SEGyges 3y ago
- arthurcolle 3y agoI can't believe people would do this, just share and republish copyrighted works over the internet. I'm in shock and in disbelief. Anyways... Is RedPajama 30T and The Pile "all you need" ? ;)
- artninja1988 3y agoThere is currently a project going on to create the pile v2 which has only permissively licensed data, because of all the bickering about copyright.
- jeffrallen 3y ago> because authors prefer to be paid for their labor FTFY.
- evilduck 3y agoI asked an AI tool to create a cheery poem about ringworms infecting kids from the 1600s and it created something that's never existed before. Which author gets paid for this labor they performed?
- JetSpiegel 3y agoIs it labor if it automatic? Which worker gets paid when I turn on my oven to heat my food? I can see value, but no labor.
- zettabomb 3y agoThis is pretty reductive - "FTFY" is rarely the witty response you think it is.
- ben_w 3y agoNaturally, but I wonder what writers are going to do when the AI trained purely on suitably licensed content is still good enough to make most redundant. (The authors in best seller's lists may well be immune for a bit longer than others writers, as they're necessarily the top 0.1% of writers, but not forever: nay-sayers claimed that AI could never beat humans at chess or go because the games required special human insight).
- mjtechguy 3y agoWould be interested to see what is in there. Luckily noone has posted the magnet link on Twitter.
- SEGyges 3y agoThe counterparties on related legal action are sufficiently litigious that it is probably smarter to DM the magnet link.
- Ninjinka 3y agoI raised a concern about the inclusion of books3 in the Pile back in 2020, and this is what the head of Eleuther (Stella Biderman) told me: "So here’s the big picture. There are three sets of datasets: 1. Data exists out there in the world. It has been collected into datasets and posted online. I’ll call this raw data. 2. We take that data, clean it, and process it for language modeling. I’ll call this per-set data. 3. We combine those per-set data into one massive dataset, the Pile. This is heavily processed, including weighing the components. We created 2 and 3 and put them online. We put 2 online so that people can reweigh and remix the data if they wish, but we expect most people to just download 3 and use it out of the box. Access to 3 will be provided in several forms, including HuggingFace and from our website. 2 and 3 are not copyright violations, even if the data is copyrighted, because they fall under fair use (at least in the US). The Pile contains code that turns 1 into 2 and code that turns 2 into 3. When you download Maroon 5 from a website, you are creating a dataset corresponding to 2. That can be copyright violation depending on what you do with it, but our use is not a copyright violation."
- anticensor 3y agoIn Europe, 2 and 3 are subject to compilation copyright and database rights.
- artninja1988 3y agoHopefully that is correct. The pile has been very valuable for open model work. It's a really high quality dataset
- D-Se 3y ago[flagged]
- layer8 3y agoI don’t understand how this can be true if set 2 contains a complete copyrighted work (say, a book) that the copyright owner hasn’t approved for such distribution. Unless I misunderstand and the “process[ing] for language modeling” is an entirely irreversible process.
- deleted 3y ago[deleted]
- fddrdplktrew 3y ago825gb seems really small
- _obviously 3y agoSeems kind of small tbqh.
- beiller 3y agoIt seems small, until you try to download it.
- jMyles 3y agoSo much of this thread is concerned not with the achievement of this data set, but with the (by comparison) silly and outdated spat over how to frame it as "property" for the purposes of government intervention (pursuant to which jurisdiction?). The era of intellectual "property" is over. Let's be at peace with that and just move on into the next age.
- __lbracket__ 3y agoLLM are of use to megacorps. Megacorps assume authors, painters, etc are poor and powerless (which lets face it, they are) we can b** and moan on HN, but megacorps will find ways to use copyrighted works for free.
- quatrefoil 3y agoWhile a lot of attention has been given to books3, another large component of this dataset is the deceptively-named "OpenWebText2". What's that? It's a scrape of 15 years' worth of third-party websites that were linked to from upvoted Reddit submissions. I know this includes some of my writing.
- 7moritz7 3y agoCare to give me your domain name so I can check all major llms for plagiarism? I have a feeling none of them can produce a sentence from your writings
- quatrefoil 3y agoIt takes deliberate effort, but I was actually able to get pieces of my writing out of one of the leading LLMs (not ChatGPT). This is not particularly unique, a number of folks demonstrated the same.
- 7moritz7 3y agoHow long were those pieces?
- fennecfoxy 3y agoI would probably ask more how unique the string of text was; can't lay claim to something where the words naturally follow one another and searching Google comes up with several results.
- observationist 3y agoRelevance and impact aside, if you publish something to the internet on a site with no access restriction in place, I don't know how you can keep a straight face while claiming some sort of moral right to the content. It's the equivalent of broadcasting it over radio, or printing and delivering it straight to the doorsteps of millions of random individuals. Methinks you doth protest too much, or something. There are ways of copyrighting data, and establishing ownership of intellectual property. Your tumblr fanfic, youtube comments, or HN discussions are not legitimate copyright avenues. Stuff you post to legally scrapeable websites are fair game for fair use. I can do anything I want in private to any data I collect. I could create an awesome HN LLM on the scraped datasets, and use it privately to my hearts content. I can even set up an API to that LLM that generates content, and, given recent rulings, even if i had all the written copyrighted data in the world, as long as I was making good faith efforts to ensure copyright was being respected and works weren't being recreated verbatim, then I could even use that model commercially. I just couldn't sell it to other people, or distribute it, without entering a different legal regime. I can collect any data I want from public facing websites. That's how the internet works; it's how it was designed. There are authentication mechanisms, network configurations, and a myriad other access control schemes you can implement to prevent public access. If you post to sites without those mechanisms, you're tacitly agreeing to give up any plausible claims of protection against a wide array of fair uses well established by precedent cases at this point. If you don't prevent public access, and you've got a domain name on a server, you're tacitly inviting the world to come download whatever it is you have on your server. This is a social good. This is what we want when we participate in the internet. Insisting on some sort of vague entitlement as to how "your" data gets used completely bypasses the fact that anything you consider to be misused in OpenWebText2 fundamentally stems from the fact that you posted the content to a publicly visible website and gave up any say in what happens thereafter. It was scraped fair and square. Don't complain that you didn't know the rules, or that life isn't fair. It's not even clear that terms of service or those little popups on public websites have any legal relevance. If your website is open to the public, then it's fair game. If you post content to a public website, then that content's fair game.
- intalentive 3y agoThis is 4 years old. Why the top of HN now?
- willvarfar 3y agoAre there any simple text editors or wysiwyg that have local LLMs and can tidy up and auto-suggest whole paras of slick verbage as you type?
- clooper 3y agoThe big Hollywood studio pay a lot of money to various cyber security companies to look for pirated content and send cease and desit letters to hosting companies for letting their users distribute copyrighted content. If authors and artists were to join a data union they could do the same thing as studios. If copyrighted law has any real teeth then the data union can send legal requests to whoever is hosting the content and requesting it to be taken down. I'm not a lawyer but I know the studios definitely do this.
- kristianp 3y agoPlease add (2020) to the title.
- deleted 3y ago[deleted]
- dang 3y agoRelated: The Pile: An 800GB dataset of diverse text for language modeling (2020) - https://news.ycombinator.com/item?id=36685115 https://news.ycombinator.com/item?id=36685115 - July 2023 (70 comments)