8 ms·
The Battle over Books3
- skilled 3y agoWhat's next for models like GPT now that a lot of sites will outright block CCBot, GPTBot and others? How big of an impact is this going to have on the LLM itself? Isn't OpenAI in a bit of a pickle in regards to this? The problem with my question is the following: Content gets syndicated anyway, so if DigitalOcean blocks GPTBot (which it does), pretty much every single one of those tutorials will be syphoned off to other sites, which are unlikely to block GPTBot themselves. How will DigitalOcean (or any other company) address this? It looks to me like it's Catch 22 in every direction you look, and unless you're someone like The New York Times who can afford to outright protect the data with licensing... It's just something thats been on my mind lately but I don't understand the finer details of it.
- ChatGTP 3y agoMy guess is the big players hope is to steal an enough content and then build a self training LLM based off synthetic content (rehashed original works) before the theft part matters. Not sure how close they are to achieving but this seems to be a common SV gamble. Steal or do something shady, raise enough money / power so by the time your noticed, you have the money to win in the courts. You already see the propaganda about “none of this should matter because the cure for cancer is on the way courtesy of AI.” I mean maybe it it but it smells fishy to me.
- p1esk 3y agoAn LLM can be trained to find relevant knowledge online. It doesn’t have to be trained on the entirety of all existing knowledge.
- ChatGTP 3y agoGood luck when everything is paywalled
- Sindisil 3y agoGiven the already huge cost of training, and the evident lack of concern the LLM folks seem to have for copyright, why wouldn't the AI groups purchase subs to scrape the paywalled content? The would possibly need to apply some effort to appear human, but that should only throttle the rate, not stop their scraping all together.
- ChatGTP 3y agoIt's more difficult to scrape pay walled content no? Clearly, places like Reddit have wised up to this and are making API usages non-free for example, so while it's not impossible, you can see the limitations being put into place already. Twitter is another one. It seems like all this data is now considered gold and people lock up gold?
- visarga 3y ago> An LLM can be trained to find relevant knowledge online. Why do you think chatGPT lost its Web Search plugin lately? Copyright lawsuits. You can't even use copyrighted content in the prompt because it will make the model makers liable.
- p1esk 3y agoBut this doesn’t make sense - how is using ChatGPT to find information different from using a search engine? Especially if ChatGPT clearly lists its sources?
- chasing 3y agoLaws. And the lawsuits that arise when such theft is discovered.
- capableweb 3y agoEasy to work around. Contract someone outside of the jurisdiction to provide a dataset, then it's up to them to deliver it to you. You "weren't aware" of the data source until people outside the organization starts shouting or the police starts asking questions, but the model "doesn't contain any of the data" so you continue shipping the model/product.
- pk-protect-ai 3y agoYou will need to train the model in the same jurisdiction too, to avoid any kind of intervention into the training process. Ideally that would be a shelter company that "sells" you "training services" in that jurisdiction.
- thechao 3y agoI don't know a lot of judges, but shenanigans like that (maybe the second or third time) are a great way to get a summary judgement, and find your way to county for a few days to think about the intent of a law.
- crote 3y agoWho is going to enforce those laws? Big publishers are more than happy to settle with AI companies - they just their slice of the pie after all. But who is going to protect, say, your Hacker News comments? Are you going to sue the AI company? Is YC going to sue? Are you going to sue YC for not banning crawlers in their robots.txt?
- dale_glass 3y agoWhy would I even bother doing that? I just write comments, it's not some ineffable wisdom. I'm writing them in public. I don't expect to profit from them somehow. Heck, collecting whatever cents I might be owed for being a drop in the ocean is a losing move in my country.
- drubio 3y agoIt's going to be a walled garden dystopia, with everyone and everything asking to sign-up, subscribe or whatever to monetize content. Scraping & siphoning content, ad blockers, both sides have been in arms race for years, AI and LLMs will just be the last straw, before almost anything of value is behind a pay/subscription wall.
- civilitty 3y agoIs all the data on the internet from 2010 to 2025 that much more valuable than all the data on the internet from 2010 to 2022? Better data yes, but more data up to the present day? Do we really need that to keep improving AI? GPT is already capable of incredible generalized language understanding and I'd wager we've long since hit diminishing returns from raw internet data. RLHF, fine tuning, and better (and more data efficient) architectures are what we need now.
- radarsat1 3y agoPeople have somehow conflated artificial intelligence with "oracle that knows everything" and so keeping models up to date with recent knowledge has become a must. Of course, that could be done by fine tuning techniques and better architectures that can outsource knowledge retrieval to tools, all very interesting ongoing work on that topic, but even these approaches require to not be "blocked" in their data retrieval tasks for them to work well. I'm sure you can do lots of interesting research using outdated datasets but for companies creating products this will not be sufficient.
- photonerd 3y agoThe “somehow” being “it was explicitly marketed that way”
- rhdunn 3y agoLanguage shifts, so NLP models need to understand those. For example, compare the use of "gay" before ~1980 and after. Some words change in spelling, like "to-morrow" present in works around 1800 changing to "tomorrow" in current usage. Some words are also coined, like "woke" or "enshitification". If your data or models don't account for those then it can make mistakes. For example, if a model is only trained on modern sources (and does not know about Early Modern English 2nd person pronouns "thy"/"thine"/etc.) then it can easily get confused when determining parts of speech, which then affects other down-stream processing.
- cj 3y ago
- meiraleal 3y agoSPAs being difficult to crawl is a feature, after all! :)
- blibble 3y agoI did a random sample of some news sites across the political spectrum: every single one I checked, other than ft.com blocks GPTBot
- Aurornis 3y agoDoes anyone have more content about what makes Books3 so special relative to Bibliotik? Was it processed somehow, or just compiled into a single file? I feel like I’m missing something from this and all of the other articles about Books3. It sounds like he downloaded all of the books from a book piracy site then rehosted them with the “Books3” name. Surely there must be more to the story? Or is the story simply that a professor hosted pirated content under his own name under the guise of AI training? This is the kind of effort that could have been done anonymously, just as all of the pirated books had already been uploaded and hosted anonymously. I’m not sure why he expected any different outcome by re-pirating everything under his own name. The journalists seem to be loving it, though. All of the tech journals have an article about this guy.
- dragonwriter 3y ago> Does anyone have more content about what makes Books3 so special relative to Bibliotik? They are essentially the same content (that is, the documentation for the copy of books3 seoarately hosted in huggingface says that it is all of Bibliotik in plaintext form, presumably as of a particular point in time.) > This is the kind of effort that could have been done anonymously Sure, its something each group training an AI could do independently at greater aggregate cost until someone succeeds in taking the original source down, but not only would that be costlier, but it in would involve less transparency and comparability across model architecture, or at least required the transparent, comparable trained version to be different from the full version.
- Aurornis 3y ago> Sure, its something each group training an AI could do independently at greater aggregate cost until someone succeeds in taking the original source down, but not only would that be costlier, but it in would involve less transparency and comparability across model architecture, or at least required the transparent, comparable trained version to be different from the full version. I meant he could have uploaded it under a pseudonym rather than broadcasting to the world that he was the one doing the uploading.
- Michelangelo11 3y ago
- gremlinsinc 3y agoI lean to the side of AI progress over copyright, though I fear AI becoming stupid when incentive for original works maybe goes down, so I think society needs to figure out some social contract at least, like subsidies to keep new content flowing. For instance blogs will stop posting if chatbots in search grab all the answers directly from their sites bypassing all ad revenue.
- intrasight 3y ago> incentive for original works maybe goes down I don't know why you would have that concern. LOTS of people write because they just want to share. Now broaden it out to include speech. LOTS of people talk - it's what we do. Progress in AI will happen because humans like to express themselves. The challenge isn't copyright. It's figuring out how to capture the vast content that just isn't getting captured. Also, this is really only an issue for "new intelligence" - if you really think there is such a thing. Personally, I do not. I think like 99% of all human intelligence is in the out of copyright corpus.
- OfSanguineFire 3y ago> LOTS of people write because they just want to share. The blogging scene from the early millennium is now a shadow of its former self, and one of the most often stated reasons for abandoning blogging is “my site just wasn’t getting many views any more”. In a world where AI generated content abounds, there will be even fewer eyeballs on whatever one shares and therefore less feeling of reward for sharing. Moreover, the people still blogging are often loading their content with referral links, because in an economy full of glamorous influencers, even ordinary people are tempted to seek some financial reward for sharing beyond the mere pleasure of it. Less eyeballs due to AI competition means fewer people clicking those referral links.
- sdenton4 3y agoYah, blogs are already a rounding error in the corpus, and that has nothing to do with llms. Those of us who are still blogging are already doing it in spite of ~waves hand broadly at the world~. I'm not truly sure that llms mean less eyeballs, though. They produce mediocre content in an arena where high quality content matters. There's already a massive pile of crap on the internet; it's already all about surfacing the relevant and the interesting bits.
- rhdunn 3y agoI wonder how this affects things like the Oxford English Dictionary, wiktionary, and other corpus-based linguistics, which rely on sample sentences of the word usage in order to determine the context. Because language is an evolving thing, it is almost certain that they have referenced sentences from copyrighted sources. E.g. I'm willing to bet that they have the sentence where Cory Doctorow introduces the term "enshittification". (The OALD 7ed in it's foreword even states "Corpus analysis now makes it possible to draw authentic examples from a vast range of attested contemporary usage. A concordance will display hundreds or thousands of them to choose from.") I suspect that the inclusion of a few sentences -- especially those that introduce a new word or usage of a word -- are fair use, but the inclusion of the entire texts is not. This then brings up an interesting point where the computer scientists/linguists developing tools like WordNet or other NLP databases would be at an advantage to those that take the approach of throwing a lot of data into a neural network and hoping for the best. Yes, it is a lot more work/effort to develop those NLP databases, but in the end they may end up being more robust, especially around the question of copyright.
- PeterisP 3y agoIt shouldn't affect corpus linguistics but the other way around, because all these things have long ago (even before computers) been legally contested by authors and publishers w.r.t. what can be done to text by dictionary makers and corpora managers without the authors' permission, so unless new laws get passed, the current precedents establishing what's permissible for corpus linguistics would still be valid and also be relevant for treatment of machine learning models. In essence, the long established principles for text analysis is that facts about text (concordances, collocation statistics, n-gram counts) are neither copyrightable nor derived work, and thus can be calculated, gathered, used and distributed even if copyright holders of the source data object. Now a court might judge that training a large language model is substantially different or that it's effectively the same, but such a decision wouldn't affect corpus linguistics and how they use sample sentences, only whether LLMs get the same treatment or not.
- rhdunn 3y agoInteresting, thanks!
- bagswatchesus 3y ago[dead]
- fbdab103 3y agoFrom a preservation angle, how big is Books3? Is it easy enough for mortals to mirror for that unlikely future where it might be possible to self-train reasonably good models from scratch if provisioned with data?
- bvm 3y ago800gb
- CamperBob2 3y agoWhere are you getting that figure?
- jwitthuhn 3y agoThat is the size of 'The Pile' which contains books3 among many other things.
- banana_giraffe 3y agobooks3.tar.gz itself is ~37gb compressed. Often really the entire "The Pile" dataset (composed of both the mostly compressed archives, along with a ~450gb compressed jsonl compilation of the data) is being discussed. That's around 825gb.
- Tokumei-no-hito 3y agoOut of curiosity why is jsonl so popular in the ML space?
- fbdab103 3y agoNot speaking for ML, but I love jsonl as a distribution format. Lots of storage overhead vs a more appropriate bulk container, but it makes it trivial to stream, sample, concatenate, etc. Anything else (eg csv or parquet) is going to require better tooling and/or just a little shell magic to handle headers.
- DanHulton 3y agoAny time I see the phrase "democratize access," my spidey-sense starts tingling. It's almost never used to describe an action that's an unadulterated good for society. It's USUALLY used to describe something sketchy at best, or even outright evil, with the justification that only "the bad guys" have access currently, and everything would be better if EVERYONE had access. Look, I get that unethical corporations using this pirated training data for their artist-usurpation machines is bad, right? But EVERYONE being able to dismiss the rights and wishes of current artists while they work to create artist-usurpation machines of their own? That's not any better! You don't need to "democratize access" to that!
- lewhoo 3y agoThat's how I see it as well. To democratize art, music etc. now means to remove the skill component with the usage of all the work done so far. No one is actually prevented from pursuing those things and if you don't want your art to become training data you're ruining democratization and are somehow against the will of the people.
- __loam 3y agoThe level of disrespect for artists I've seen here and on other forums with regards to this technology has been staggering. The entitlement for work you didn't do is so gross.
- labster 3y agoAs a fanfiction writer, I find the entitlement of authors and corporations over work that I created gross. I transformed their work into something else, advertising the original work and making a bigger market for it, and in exchange we get C&Ds and copyright claims. Someone writes a reskin of The Odyssey and then a corporation claims every derivative for 75 years.
- __loam 3y agoI have a lot more sympathy for you than the corporate interests that are currently trying to profit off of your work without your knowledge.
- msoad 3y agoit would be a shame if it would be possible to train a very useful AI using all the books in the world but corporate greed wouldn’t allow it
- primordialsoup 3y agoIt's pretty much the opposite: Big corporates are the ones training the models (and benefiting from it).
- CamperBob2 3y agoEither way, it has to happen, and it will happen. It might as well happen on universally-equitable terms.
- artninja1988 3y agoOther big corporation hold a large portion of copyrights trained on as well. I tend to side with everyone being able to do it
- microtherion 3y agoEverybody with a few million dollars to spend per training run of their model in pure compute cost…
- edent 3y agoWhat if I don't want my book used for that purpose? It isn't as though the AI companies have even paid for a single copy of the authors' books.
- crote 3y agoWhat worries me most is that this is likely to only increase the gap between large corporate creator and small independent creators. Large AI companies probably are not too worried about making deals with corporate content creators. Having access to content from a trigger-happy creator is only going to increase their advantage over competitors, after all. And if those creators were instead to try to introduce legislation, the AI companies would risk losing access to content from small creators without the means to sue too. We seem to be moving into a world where corporate content cannot in any way be reused, remixed, or even archived. You cannot even own a copy - it is only accessible for a monthly fee and can disappear at any time. Meanwhile, anything created by independent creators is fair game to steal and rip off. Copyright was intended to promote and protect human creativity, but instead we got a rent-seeking mechanism used to stifle original creation.
- EGreg 3y agoHow shocking! A monopoly right granted by the government to exclude others from use of an idea (patents) and creative works (intellectual property) was intended to help the little guy, but ended up helping the big boys extract rents instead, while exploiting the little guy because they sold their rights to survive and get access to a platform? https://janefriedman.com/i-would-rather-see-my-books-pirated/ https://janefriedman.com/i-would-rather-see-my-books-pirated... It’s almost as if governments (even capitalist ones) work with industry and concentration of power perpetuates this kind of consolidation further. They keep us distracted so we don’t have enough collective willpower to get together and demand reform, or even better — create our own alternative open ecosystems. I write about many other examples of government-industry distracting us here: https://magarshak.com/blog/?p=362 https://magarshak.com/blog/?p=362
- gochi 3y ago[flagged]
- lewhoo 3y agoHow so ? Copyright was introduced along with the print press to protect the works of authors. When you buy a copy of a book it's not exactly private property of the author that you buy, is it ?
- yieldcrv 3y ago> “It is only fair that you compensate us for using our writings, without which AI would be banal and extremely limited,” the letter states. These models are going to come out so fast from one-off contracts. Its not the line that some creatives think it is. Its a 2 year delay at best.
- JKCalhoun 3y agoInteresting. When it's copyrighted works in a digital form (plain text, ePub, whatever) it's a legal issue. ROT13 the text is the data still a copyright violation? I imagine so since it's a trivial thing to restore it to a legally volatile form. I understand that an unresolved issue is whether, once ingested into an LLM, the trained LLM is in violation of copyright. One wonders if human readers too are in violation of copyright for having been "trained" as well when they read a book. Is a "brain transplant" from one LLM to another a thing? Perhaps just a trivial copy of the node weights or whatever they're called. That would would not let the target LLM off the hook with regard to copyright violation I expect. But what if one LLM "taught" another. Maybe that is not a thing yet.
- the8472 3y ago> Is a "brain transplant" from one LLM to another a thing? At least it's a thing for vision models and was also used to train OpenAI Five (the dota AI)[0]. So it probably applies to LLMs too. [0] https://cdn.openai.com/dota-2.pdf https://cdn.openai.com/dota-2.pdf (page 2)
- walnutclosefarm 3y agoIt would be interesting to see someone dig into the differences between the capture of language and culture, and control over how it gets fed back into cultural discourse, that we fear corporations achieving with commercialized LLMs, and the control pre-internet publishers had (and to a certain extent, still do), over what content, and in what format, language was disbursed and distributed. I am not at all certain that having your work used to train LLMs is a bigger threat to writers ownership of their work than publishing houses were and are. I also suspect that if we had an effective mechanism to prevent use of copyrighted work in training, it would necessarily behoove an artist to opt their content out. Will you really want to be excluded what may well become the canonical mechanism for searching and generating language?
- fiforpg 3y ago“I was poking around, Googling ‘how to download Library Genesis,’” LG literally has this info on its front page. Researchers will research, I suppose.
- sillysaurusx 3y agoMore to the point, it’s huge. 33 TB is a massive amount, and I was grateful to sidestep it by downloading just the epubs. Still, there’s always room for books4.
- fsckboy 3y agois there enough data on the web for a LLM to be agnostic about the source languages, Russian, Chinese, English, German, etc? Where training on Russian and Chinese and English and German etc sources would also incorporate enough information about translation that if the AI learned about some topic only through Chinese sources, it could still recognize/use/apply/express those ideas in English?
- sillysaurusx 3y agoThat’s a fascinating question. I’m not sure. It seems like if it learned about a topic in Chinese, it would be able to express it in English, but I haven’t seen this tested.
- deleted 3y ago[deleted]
- BashiBazouk 3y agoBooks3 is the easy/fast/cheap method but if the quality of the model really brings in some sort of revenue is there anything stopping a company from buying/checking out of the library all these books, scanning and ORC'ing them and adding to the model the hard way?
- balthigor 3y agoSee for yourself it's worth the fuss. magnet:?xt=urn:btih:bu3ganlgj7pvdt56t5ztsu52gjlxnzth&dn=EleutherAI_ThePile_v1&xl=772891257239&fc=51
- caesil 3y ago>He sees the widespread practice of training AI on copyrighted data as outrageous, and finds it infuriating that this behavior gets defended with claims that it’s democratizing access to information. “Open source doesn’t mean you took a bunch of people’s shit and gave it away for free,” he says. “That's theft.” >Whether the defendant had purchased a signed copy or flagrantly shoplifted a dog-eared paperback wouldn’t matter during arguments over whether The Bedwetter, Too was a derivative rip-off or a transformative parody. This strikes at the heart of why this case is about to be laughed out of court. The argument the plaintiffs are making is that ChatGPT is a "derivative work", i.e. letting people use the software is akin to distributing carbon copies of the book at issue, with at most slight modifications (typical derivative works include translations, screenplay adaptations, etc.). Since ChatGPT obviously cannot literally produce the full text of the book on command, the very strained position they're trying to advance is that short, several-paragraph summaries constitute a derivative work. That is to say, they're arguing that writing, say, a review, or a book report, is an act of copyright infringement tantamount to taking a book, translating it into Japanese, and selling that translation. It's a deeply stupid and wrongheaded argument, and it deserves to die a quick death.
- HWR_14 3y agoWriting a review or a book report is very much creating a derivative work in copyright law. Copyright law then says that these derivative works are a fair use. It does not follow that other derivative works that you personally feel are as serious are also fair use.
- deleted 3y ago[deleted]
- nostril 3y agoIf anyone wants to download Books3, it seems to be available here: https://thenose.cc/public/AI/EleutherAI_ThePile_v1/pile_preliminary_components/ https://thenose.cc/public/AI/EleutherAI_ThePile_v1/pile_prel... (We noticed The Pile was recently taken offline, so we hosted it: https://thenose.cc https://thenose.cc. Apparently Books3 was also a part of The Pile, so feel free to download.)