6 ms·

AI Books4 Dataset for training LLMs further

46 points by ImperiumOfMan 2y ago

mannyv 2y ago
It's strange that you could actually use these datasets to poison other people's LLMs - since there's no way you could go through everything and vet all the data in the dataset.
__forward__ 2y ago
The copyright situation around all this is very... interesting. Pretty clear that this dataset is not legal but what about resulting models? What if the texts actually where bought 'properly'?
- sp332 2y ago
  Buying a copy of the book would not give you any copyright license. You could only make copies for personal use.
  - CaptainFever 2y ago
    If you are in a jurisdiction with TDM exceptions, buying a personal copy does allow you to train on it.
- DeathArrow 2y ago
  If you steal a PC and use it to build a very successful app, would that app be legal? Would the use of the said app by third parties be legal?
- chasd00 2y ago
  The race is on to figure out a way to get LLMs to produce content to be used for training other LLMs in a satisfactory way. Eventually the dataset question will get figured out in the courts but if there’s a technique to generate more training data in an automated way then the court decision doesn’t matter. Edit: also, I don’t believe court decisions can be enforced retroactively so existing LLMs would be safe but I’m most definitely not a lawyer.
zaptheimpaler 2y ago
Very cool. Guess you can just pirate 400K books stealing millions from authors as long as its a "dataset for LLMs" instead of a regular old piracy dump. No doubt there are books in there that took decades of experience to write and knowledge that took a lifetime to learn. Stealing is just OK as long as its for AI lol, and as we devalue writing we guarantee the quality of writing will continue to degrade.
- freefaler 2y ago
  Hmm... but the whole internet is also indexed by Google and they make a lot of money out of the content. Should they pay too? If this was enforced would there be search engines?
  - sp332 2y ago
    It's different for websites because they are intentionally served up for free. Google had a very hard time getting a license to even index book contents. And there's an ongoing tension with paywalled websites. They do pay for some news sites, in the Google News Showcase.
  - hawski 2y ago
    One of the reasons behind the backlash against Google's AMP was that they would contain the full experience. You are on a Google's OS (at least significant amount of users are), on a Google browser, using a Google search service and from there you go to Google servers to get the content. The main difference is the last step.
- hawski 2y ago
  The Pirate Bay should consider rebranding as The Dataset Bay. I am sure there are wild times ahead. At some time it will become outlawed in the way regular piracy is. Not because of book authors or publishers, but either because of Hollywood or AI safety being used by bigtech to limit competition.
- Workaccount2 2y ago
  I don't know if there is a lot of confusion, purposely misleading deception, or both, but I cannot grasp the argument for "training on copyrighted data is a violation of copyright". What I can grasp is "output of copyrighted data is violation of copyright", and the fix seems to be a straight forward dumb software contentID filter on the output.
DeathArrow 2y ago
It's kind of a grey area. It's sci-hub allowing to read articles for free a bad thing? Is it legal? There's not always going to be a correspondence between what is wrong from a legal pov and what is wrong from a moral pov.
- bayindirh 2y ago
  How many books you can read and remember exactly as is, plus reproduce remix the text as is? What's this number for an LLM?
closetkantian 2y ago
Can anyone me one tell me how big this is in GB?
- simongr3dal 2y ago
  Around 250GB, several terabytes once uncompressed I assume.
- crtasm 2y ago
  libstc2.jsonl.zst 231 GB
yareal 2y ago
This is simply piracy. This reads exactly like a scene release of any pirated material.
- harshreality 2y ago
  Scene releases are in human-consumable formats. Books as lines of text are not human-consumable (by reasonable humans). There's a strong case for sharing this being fair use, even if distributing hundreds of thousands of ebooks would ordinarily violate copyright. I don't know what OpenAI's training corpus is or how they got it, but in case anyone has forgotten, Google has its own license-free collection of scanned books. It had already run that through OCR (and probably periodically re-OCRs) to generate its google books indexes, so there is no copyright argument against Google's book-trained AI models (assuming they train on books). An argument that Books4 is piracy is an argument for an oligopoly on good AI models.
  - bayindirh 2y ago
    > There's a strong case for sharing this being fair use, even if distributing hundreds of thousands of ebooks would ordinarily violate copyright. If you're using the resulting model trained with this for (academic) research (independently or at a university), that's true. If you're earning money with the model (cough chatGPT, Bard, Gemini, et. al cough), that's false. Edit: Original version of the above paragraph started as "If you're doing...", making the below comment true. Edit is made to clarify the point further.
    - harshreality 2y ago
      If you're complaining about the training, you're confusing the entity doing the sharing in this case, with the entities doing the training. ETA: If you're complaining about use of the model, this becomes a dispute over how close to an original work an output can get before it becomes a copyright violation. That seems unresolvable. Look at court cases where musicians sue over short sequences of notes. Nobody knows where the threshold should be for that, for poetry, for flash fiction, for academic papers, for novels, for paintings, or for voice samples. Courts just go with whatever they think feels right in a particular case. That is an untenable legal situation in a world with AI models like these. Also, to reiterate, Google has its own fulltext, license-free corpus of books, which it undoubtedly has used to train models. What's your view on that? Should Google be the only entity allowed to train on large books datasets because it happened to get one through a loophole of collaborating with libraries to OCR their collections?
fancyfredbot 2y ago
Where did this come from? Even if assembled from multiple sources the full text for 400k books and 6M other publications would be very hard to get. Did someone hack an e-book store? Or somehow extract data from Google books?
- freefaler 2y ago
  From all the e-books available on the webs... Check annas-archive.org for example.
  - fancyfredbot 2y ago
    Wow so this is actually just a tiny subset of what was already openly available. I'm still interested in how all of this was gathered together though - looks like Anna's Archive got the books from other large datasets, but I'm still confused about the ultimate origin of the data.
    - ImperiumOfMan 2y ago
      Origin of the data: - LibGen collections gathered over decades - Sci-Hub - Additional fresh scholar papers and books collected by Library STC
    - freefaler 2y ago
      the main ebooks sources - bought/library borrowed books with DRM removed and shared - scanned & OCRed books from archive.org & related projects - non-DRM bough & shared books
    - michaelnny 2y ago
      From my experience as an author of a tech book, the ebook is pirated even before the official publication date, the authors only have access to the ebooks fees days after the publication date. That’s why I’ll never write a book again.
      - fancyfredbot 2y ago
bnj 2y ago
That’s some really nice intellectual property they have there. Be a shame if someone thought about it.
deleted 2y ago
[deleted]
squigz 2y ago
> [removed] https://web.archive.org/web/20240519104217/https://old.reddit.com/r/datasets/comments/1cvi151/ai_books4_dataset_for_training_llms_further/ https://web.archive.org/web/20240519104217/https://old.reddi... magnet:?xt=urn:btih:a904e660355c49006b2e7d43893d31bf3c2be9cc&dn=libstc2.jsonl.zst&tr=udp://tracker.opentrackr.org:1337/announce&tr=https://tracker1.ctix.cn:443/announce&tr=udp://open.demonii.com:1337/announce https://tracker1.ctix.cn:443/announce&tr=udp://open.demonii.... > More than 400,000 fiction and non-fiction book full-texts. Multiple languages, curated, deduplicated. > More than 6,000,000 scholarly publications, magazines, and manuals full-texts. Multiple languages, curated, deduplicated. > 150,000,000 metadata records

46 points by ImperiumOfMan 2y ago

It's strange that you could actually use these datasets to poison other people's LLMs - since there's no way you could go through everything and vet all the data in the dataset.

__forward__ 2y ago

The copyright situation around all this is very... interesting. Pretty clear that this dataset is not legal but what about resulting models? What if the texts actually where bought 'properly'?

sp332 2y ago
Buying a copy of the book would not give you any copyright license. You could only make copies for personal use.
- CaptainFever 2y ago
  If you are in a jurisdiction with TDM exceptions, buying a personal copy does allow you to train on it.
DeathArrow 2y ago
If you steal a PC and use it to build a very successful app, would that app be legal? Would the use of the said app by third parties be legal?
chasd00 2y ago
The race is on to figure out a way to get LLMs to produce content to be used for training other LLMs in a satisfactory way. Eventually the dataset question will get figured out in the courts but if there’s a technique to generate more training data in an automated way then the court decision doesn’t matter. Edit: also, I don’t believe court decisions can be enforced retroactively so existing LLMs would be safe but I’m most definitely not a lawyer.

zaptheimpaler 2y ago

Very cool. Guess you can just pirate 400K books stealing millions from authors as long as its a "dataset for LLMs" instead of a regular old piracy dump. No doubt there are books in there that took decades of experience to write and knowledge that took a lifetime to learn. Stealing is just OK as long as its for AI lol, and as we devalue writing we guarantee the quality of writing will continue to degrade.

freefaler 2y ago
Hmm... but the whole internet is also indexed by Google and they make a lot of money out of the content. Should they pay too? If this was enforced would there be search engines?
- sp332 2y ago
  It's different for websites because they are intentionally served up for free. Google had a very hard time getting a license to even index book contents. And there's an ongoing tension with paywalled websites. They do pay for some news sites, in the Google News Showcase.
- hawski 2y ago
  One of the reasons behind the backlash against Google's AMP was that they would contain the full experience. You are on a Google's OS (at least significant amount of users are), on a Google browser, using a Google search service and from there you go to Google servers to get the content. The main difference is the last step.
hawski 2y ago
The Pirate Bay should consider rebranding as The Dataset Bay. I am sure there are wild times ahead. At some time it will become outlawed in the way regular piracy is. Not because of book authors or publishers, but either because of Hollywood or AI safety being used by bigtech to limit competition.
Workaccount2 2y ago
I don't know if there is a lot of confusion, purposely misleading deception, or both, but I cannot grasp the argument for "training on copyrighted data is a violation of copyright". What I can grasp is "output of copyrighted data is violation of copyright", and the fix seems to be a straight forward dumb software contentID filter on the output.

DeathArrow 2y ago

It's kind of a grey area. It's sci-hub allowing to read articles for free a bad thing? Is it legal? There's not always going to be a correspondence between what is wrong from a legal pov and what is wrong from a moral pov.

bayindirh 2y ago
How many books you can read and remember exactly as is, plus reproduce remix the text as is? What's this number for an LLM?

closetkantian 2y ago

Can anyone me one tell me how big this is in GB?

simongr3dal 2y ago
Around 250GB, several terabytes once uncompressed I assume.
crtasm 2y ago
libstc2.jsonl.zst 231 GB

yareal 2y ago

This is simply piracy. This reads exactly like a scene release of any pirated material.

harshreality 2y ago
Scene releases are in human-consumable formats. Books as lines of text are not human-consumable (by reasonable humans). There's a strong case for sharing this being fair use, even if distributing hundreds of thousands of ebooks would ordinarily violate copyright. I don't know what OpenAI's training corpus is or how they got it, but in case anyone has forgotten, Google has its own license-free collection of scanned books. It had already run that through OCR (and probably periodically re-OCRs) to generate its google books indexes, so there is no copyright argument against Google's book-trained AI models (assuming they train on books). An argument that Books4 is piracy is an argument for an oligopoly on good AI models.
- bayindirh 2y ago
  > There's a strong case for sharing this being fair use, even if distributing hundreds of thousands of ebooks would ordinarily violate copyright. If you're using the resulting model trained with this for (academic) research (independently or at a university), that's true. If you're earning money with the model (cough chatGPT, Bard, Gemini, et. al cough), that's false. Edit: Original version of the above paragraph started as "If you're doing...", making the below comment true. Edit is made to clarify the point further.
  - harshreality 2y ago
    If you're complaining about the training, you're confusing the entity doing the sharing in this case, with the entities doing the training. ETA: If you're complaining about use of the model, this becomes a dispute over how close to an original work an output can get before it becomes a copyright violation. That seems unresolvable. Look at court cases where musicians sue over short sequences of notes. Nobody knows where the threshold should be for that, for poetry, for flash fiction, for academic papers, for novels, for paintings, or for voice samples. Courts just go with whatever they think feels right in a particular case. That is an untenable legal situation in a world with AI models like these. Also, to reiterate, Google has its own fulltext, license-free corpus of books, which it undoubtedly has used to train models. What's your view on that? Should Google be the only entity allowed to train on large books datasets because it happened to get one through a loophole of collaborating with libraries to OCR their collections?

fancyfredbot 2y ago

Where did this come from? Even if assembled from multiple sources the full text for 400k books and 6M other publications would be very hard to get. Did someone hack an e-book store? Or somehow extract data from Google books?

freefaler 2y ago
From all the e-books available on the webs... Check annas-archive.org for example.
- fancyfredbot 2y ago
  Wow so this is actually just a tiny subset of what was already openly available. I'm still interested in how all of this was gathered together though - looks like Anna's Archive got the books from other large datasets, but I'm still confused about the ultimate origin of the data.
  - ImperiumOfMan 2y ago
    Origin of the data: - LibGen collections gathered over decades - Sci-Hub - Additional fresh scholar papers and books collected by Library STC
  - freefaler 2y ago
    the main ebooks sources - bought/library borrowed books with DRM removed and shared - scanned & OCRed books from archive.org & related projects - non-DRM bough & shared books
  - michaelnny 2y ago
    From my experience as an author of a tech book, the ebook is pirated even before the official publication date, the authors only have access to the ebooks fees days after the publication date. That’s why I’ll never write a book again.
    - fancyfredbot 2y ago

bnj 2y ago

That’s some really nice intellectual property they have there. Be a shame if someone thought about it.

deleted 2y ago

[deleted]

squigz 2y ago

> [removed] https://web.archive.org/web/20240519104217/https://old.reddit.com/r/datasets/comments/1cvi151/ai_books4_dataset_for_training_llms_further/ https://web.archive.org/web/20240519104217/https://old.reddi... magnet:?xt=urn:btih:a904e660355c49006b2e7d43893d31bf3c2be9cc&dn=libstc2.jsonl.zst&tr=udp://tracker.opentrackr.org:1337/announce&tr=https://tracker1.ctix.cn:443/announce&tr=udp://open.demonii.com:1337/announce https://tracker1.ctix.cn:443/announce&tr=udp://open.demonii.... > More than 400,000 fiction and non-fiction book full-texts. Multiple languages, curated, deduplicated. > More than 6,000,000 scholarly publications, magazines, and manuals full-texts. Multiple languages, curated, deduplicated. > 150,000,000 metadata records