6 ms·

Searchable Database of the 183,000 Pirated Books Meta, et al., Used to Train AI

84 points by fortran77 3y ago

pobs 3y ago
Ironically, the article and searchable database appear to be behind a paywall…
skilled 3y ago
https://archive.ph/vAw3g https://archive.ph/vAw3g https://full-stack-search-prod.vercel.app/ https://full-stack-search-prod.vercel.app/
fiforpg 3y ago
Should we tell them about the library Genesis?
- a_wild_dandan 3y ago
  Let's tattle on the Library of Babel.
- KirillPanov 3y ago
  Isn't that exactly what "Books2" is? Presumably the AIcorps avoid calling it "libgen" because that would be just a bit too obvious, and now the journalists aren't calling it that to avoid the Streisand Effect. But frankly, I can't see how Books2 could be anything other than libgen.
carabiner 3y ago
It's time for UBI.
- ranguna 3y ago
  Universal basic income? What does that have to do with the article?
  - Grimblewald 3y ago
    I guess they hope if people have enough they won't sue for things like this but they fail to see that the people suing are rich to a degree many of us cannot comprehend. UBI won't change their behaviour one bit.
  - friend_and_foe 3y ago
    These people just shoehorn their chosen goals into every conversation.
tailspin2019 3y ago
Wow, every author I searched for is in there. I’m very bullish on technology progress and the future of generative AI, but at the same time I cannot fathom deciding to consume such a vast amount of pirated content (or even any pirated content) and then releasing a paid service off the back of it, and not expecting trouble down the line. For an open source, non-commercial project it might be controversial, but commercialising the aggregate pirated content of 180,000 stolen books is pretty bold. There is a very clear line between content that’s freely available (eg. scraping the web) and this. And even the web scraping is obviously controversial for some…
- sillysaurusx 3y ago
  I didn’t intend for it to be commercialized. It’s a research contribution, nothing more. It was also freely available via the web when I did it. I got all the books from https://the-eye.eu/public/Books https://the-eye.eu/public/Books. So it was literally web scraping. In fact, you can, too. https://the-eye.eu/public/Books/ThoseBooks/ https://the-eye.eu/public/Books/ThoseBooks/ seems to have a wide selection, though it’s an order of magnitude less than their Bibliotik scrape. It’s interesting that no one cared until recently.
  - deleted 3y ago
    [deleted]
  - abrookewood 3y ago
    Just because it was web scraping doesn't mean it was OK to take it. You knew the sources contained copyrighted material and you did it anyway. I'm not trying to attack you but I don't see how saying it was 'research' makes it OK.
  - tailspin2019 3y ago
    > I didn’t intend for it to be commercialized. It’s a research contribution, nothing more. FWIW my comment is aimed at the likes of OpenAI who are now profiting off this work. For example, every book that I own (as in paid money for) and searched for is in books3. I pay OpenAI currently for ChatGPT Pro. I feel like I should be getting a discount... I don't really have a strong opinion on the collation of material for research purposes as you did. LLMs are going to have a truly outsized impact on the world in the long term. But we do still need to consider the authors (individual creators) who are getting caught up in the crossfire of this knowledge/technology revolution.
sillysaurusx 3y ago
The real tragedy is that people had to work so hard to figure out exactly what was in the dataset. I just never got around to releasing the metadata, which is here: https://battle.shawwn.com/books3-metadata.jsonl https://battle.shawwn.com/books3-metadata.jsonl Dozens of people have told me I belong in prison for making books3. And as far as I can tell, it’s had zero economic impact on authors. If I do end up in prison, I think I’ll join the ranks of "people we look back on throughout history with morbid curiosity". There are lots of examples of people being imprisoned for things we now consider harmless. The buggery act was used to purge the Knights Templar. See also https://annas-archive.org/llm https://annas-archive.org/llm for acquiring ~20M books. But at this point you’d better be willing to risk being despised at best. At this point I don’t know why I even did anything in AI. llama-dl was DMCA’ed, books3 is prison fodder, and the big players do as they please with very little risk. Even if the lawsuits are successful, they’ll just pay out damages from VC money. I love ML for the sake of ML. I wanted to prove that individual researchers can contribute in a way that matters. Hopefully they’ll continue to be able to. By the way, the reporter here, Alex Reisner, is a wonderful fellow, and a fellow hacker. He has a bunch of open source projects.
- pavlov 3y ago
  > “as far as I can tell, it’s had zero economic impact on authors” Why are you the right person in the world to make these decisions on the behalf of thousands of living authors? I can understand distributing old works and abandonware. Copyright protection is far too long. But I don’t understand taking the recent creative work of somebody who’s alive and trying to make a living, and packaging it so that trillion-dollar corporations have a plausible excuse to use the work without permission.
  - anothernewdude 3y ago
    Who are publishers to do so? Because they're the ones that have decided for everyone right now. There will be zero economic impact on authors because they're already being screwed by publishers.
    - 3y ago
anothernewdude 3y ago
I mean, copyright shouldn't be the infinite years it is currently. This used to belong to everyone collectively, and then publishers (not writers) decided they owned it. Another big interest coming along now to decide that now they own it? I don't care. Give people back their culture. 5 years of copyright is sufficient.
- hknmtt 3y ago
  true. when it comes to technology, enjoy your 5 years of premium production. after that, you better have a good product, or the competition will eat you alive. as for creative works like books or movies or music, i'd go with 10 because it is more subjective type of work, creative instead of inventive and not technical and usually produced by individuals and not companies.
  - rspoerri 3y ago
    Following the same logic, everything you have archived, your wealth, your gadgets, your house, your car should belong to everybody after 10 years. Or do you want that only to apply to artists who (99% of them) dont make a lot of money anyways?
    - lamp987 3y ago
      None of those are intellectual "property".
    - anothernewdude 3y ago
      So 99% percent of artists won't notice the difference? You've convinced me. This is a fantastic idea.
- abrookewood 3y ago
  I agree that copyright has gotten out of hand, but 5 years is way too short. I can't imagine that anyone is going to spend YEARS writing a book, only for it to be public domain within 5. There needs to be commercial incentive for people to put in the effort to create things.
disposition2 3y ago
I'm curious to see how this plays out and find it unfortunate that it happened. Unfortunate because, I foresee the places where this information is shared between humans for the gain of knowledge are easier targets and will more than likely be targeted. I would be curious how an author might feel about that...their information being freely shared between humans for knowledge? Curious because in my life experience, corporations don't have to play by the same rules as human beings and actually get rewarded for actions that would be admonished if an individual were to do them. So, even though corporate owned models were trained on and leverage this information for monetary gain (and not for the general idea of sharing knowledge), they will be glorified (as often seen here) for their monetization of this 'theft' and individuals will have a harder time gaining access to information. One thing is for certain about this AI revolution, it has broken the open internet.
matthaigttttt 3y ago
Matt haig

84 points by fortran77 3y ago

pobs 3y ago

Ironically, the article and searchable database appear to be behind a paywall…

https://archive.ph/vAw3g https://archive.ph/vAw3g https://full-stack-search-prod.vercel.app/ https://full-stack-search-prod.vercel.app/

fiforpg 3y ago

Should we tell them about the library Genesis?

a_wild_dandan 3y ago
Let's tattle on the Library of Babel.
KirillPanov 3y ago
Isn't that exactly what "Books2" is? Presumably the AIcorps avoid calling it "libgen" because that would be just a bit too obvious, and now the journalists aren't calling it that to avoid the Streisand Effect. But frankly, I can't see how Books2 could be anything other than libgen.

carabiner 3y ago

It's time for UBI.

ranguna 3y ago
Universal basic income? What does that have to do with the article?
- Grimblewald 3y ago
  I guess they hope if people have enough they won't sue for things like this but they fail to see that the people suing are rich to a degree many of us cannot comprehend. UBI won't change their behaviour one bit.
- friend_and_foe 3y ago
  These people just shoehorn their chosen goals into every conversation.

tailspin2019 3y ago

Wow, every author I searched for is in there. I’m very bullish on technology progress and the future of generative AI, but at the same time I cannot fathom deciding to consume such a vast amount of pirated content (or even any pirated content) and then releasing a paid service off the back of it, and not expecting trouble down the line. For an open source, non-commercial project it might be controversial, but commercialising the aggregate pirated content of 180,000 stolen books is pretty bold. There is a very clear line between content that’s freely available (eg. scraping the web) and this. And even the web scraping is obviously controversial for some…

sillysaurusx 3y ago
I didn’t intend for it to be commercialized. It’s a research contribution, nothing more. It was also freely available via the web when I did it. I got all the books from https://the-eye.eu/public/Books https://the-eye.eu/public/Books. So it was literally web scraping. In fact, you can, too. https://the-eye.eu/public/Books/ThoseBooks/ https://the-eye.eu/public/Books/ThoseBooks/ seems to have a wide selection, though it’s an order of magnitude less than their Bibliotik scrape. It’s interesting that no one cared until recently.
- deleted 3y ago
  [deleted]
- abrookewood 3y ago
  Just because it was web scraping doesn't mean it was OK to take it. You knew the sources contained copyrighted material and you did it anyway. I'm not trying to attack you but I don't see how saying it was 'research' makes it OK.
- tailspin2019 3y ago
  > I didn’t intend for it to be commercialized. It’s a research contribution, nothing more. FWIW my comment is aimed at the likes of OpenAI who are now profiting off this work. For example, every book that I own (as in paid money for) and searched for is in books3. I pay OpenAI currently for ChatGPT Pro. I feel like I should be getting a discount... I don't really have a strong opinion on the collation of material for research purposes as you did. LLMs are going to have a truly outsized impact on the world in the long term. But we do still need to consider the authors (individual creators) who are getting caught up in the crossfire of this knowledge/technology revolution.

sillysaurusx 3y ago

The real tragedy is that people had to work so hard to figure out exactly what was in the dataset. I just never got around to releasing the metadata, which is here: https://battle.shawwn.com/books3-metadata.jsonl https://battle.shawwn.com/books3-metadata.jsonl Dozens of people have told me I belong in prison for making books3. And as far as I can tell, it’s had zero economic impact on authors. If I do end up in prison, I think I’ll join the ranks of "people we look back on throughout history with morbid curiosity". There are lots of examples of people being imprisoned for things we now consider harmless. The buggery act was used to purge the Knights Templar. See also https://annas-archive.org/llm https://annas-archive.org/llm for acquiring ~20M books. But at this point you’d better be willing to risk being despised at best. At this point I don’t know why I even did anything in AI. llama-dl was DMCA’ed, books3 is prison fodder, and the big players do as they please with very little risk. Even if the lawsuits are successful, they’ll just pay out damages from VC money. I love ML for the sake of ML. I wanted to prove that individual researchers can contribute in a way that matters. Hopefully they’ll continue to be able to. By the way, the reporter here, Alex Reisner, is a wonderful fellow, and a fellow hacker. He has a bunch of open source projects.

pavlov 3y ago
> “as far as I can tell, it’s had zero economic impact on authors” Why are you the right person in the world to make these decisions on the behalf of thousands of living authors? I can understand distributing old works and abandonware. Copyright protection is far too long. But I don’t understand taking the recent creative work of somebody who’s alive and trying to make a living, and packaging it so that trillion-dollar corporations have a plausible excuse to use the work without permission.
- anothernewdude 3y ago
  Who are publishers to do so? Because they're the ones that have decided for everyone right now. There will be zero economic impact on authors because they're already being screwed by publishers.
  - 3y ago

anothernewdude 3y ago

I mean, copyright shouldn't be the infinite years it is currently. This used to belong to everyone collectively, and then publishers (not writers) decided they owned it. Another big interest coming along now to decide that now they own it? I don't care. Give people back their culture. 5 years of copyright is sufficient.

hknmtt 3y ago
true. when it comes to technology, enjoy your 5 years of premium production. after that, you better have a good product, or the competition will eat you alive. as for creative works like books or movies or music, i'd go with 10 because it is more subjective type of work, creative instead of inventive and not technical and usually produced by individuals and not companies.
- rspoerri 3y ago
  Following the same logic, everything you have archived, your wealth, your gadgets, your house, your car should belong to everybody after 10 years. Or do you want that only to apply to artists who (99% of them) dont make a lot of money anyways?
  - lamp987 3y ago
    None of those are intellectual "property".
  - anothernewdude 3y ago
    So 99% percent of artists won't notice the difference? You've convinced me. This is a fantastic idea.
abrookewood 3y ago
I agree that copyright has gotten out of hand, but 5 years is way too short. I can't imagine that anyone is going to spend YEARS writing a book, only for it to be public domain within 5. There needs to be commercial incentive for people to put in the effort to create things.

disposition2 3y ago

I'm curious to see how this plays out and find it unfortunate that it happened. Unfortunate because, I foresee the places where this information is shared between humans for the gain of knowledge are easier targets and will more than likely be targeted. I would be curious how an author might feel about that...their information being freely shared between humans for knowledge? Curious because in my life experience, corporations don't have to play by the same rules as human beings and actually get rewarded for actions that would be admonished if an individual were to do them. So, even though corporate owned models were trained on and leverage this information for monetary gain (and not for the general idea of sharing knowledge), they will be glorified (as often seen here) for their monetization of this 'theft' and individuals will have a harder time gaining access to information. One thing is for certain about this AI revolution, it has broken the open internet.

matthaigttttt 3y ago

Matt haig