7 ms·

Where do I find the license reproductions and credits/attributions for the content being distributed in this data set? Is it all in there? Are all inclusions co

Where do I find the license reproductions and credits/attributions for the content being distributed in this data set? Is it all in there? Are all inclusions compliant? Can I know? I'm open to the argument that generators built with models that consumed copyrighted data may evade copyright obligations on their output, but surely the data sets themselves are bound by any copyright on their content?

__loam 3y ago
They stole it because they think building their toys is more important than everyone else's rights to the product of their own labor.
- johndough 3y ago
  I doubt that anyone is going to download and search through over 800 TB just to find a badly formatted copy of some book that could be found much quicker on different websites with better formatting. Authors are losing fractional cents here at most.
  - gosub100 3y ago
    so just like Office Space? (paraphrasing) "We steal a fraction of a cent from each transaction, who do we hurt? Nobody. We just put the remainder into our account!" Sorry that's not how damages are calculated in the US tort system.
    - johndough 3y ago
      I do not know how damages are calculated in the US tort system. What do they say about the books3 dataset? I also think that the case is different here, since in your example, there is a specific amount of money being stolen, while in the books3 case, there is an unspecified amount of money not being made by the authors.
      - SEGyges 3y ago
        I am pretty sure if the authors were trying to license their works for this purpose we would just not use them at all; it is difficult to see under what circumstances they would stand to profit from this other than by suing people after the fact over it.
        doug_durham 3y ago
        I think you could argue that authors could profit from their works being cited in an LLM response. It could drive sales of their works much like citations do on the web. The counter argument is that and LLM could give you the Clif Notes version of the work and thus taking away a portion of sales.
        SEGyges 3y ago
        In a world where the options were to 1) pay the author, 2) implement guaranteed citation of the author any time the model gave an answer that was directly derivative, with an option to not do so if the summary was sufficiently vague, or 3) ignore the author's book completely as training data we would all choose 3).
        __loam 3y ago
        And the authors would probably be very happy that you did.
  - __loam 3y ago
    The penalty is up to $150k per violation.
    - sp332 3y ago
      For uploading, not downloading.
- pk-protect-ai 3y ago
  they have stole nothing, they make no profit from it as well.
  - idle_zealot 3y ago
    Oh, so there aren't AI companies charging for access to private models?
    - pk-protect-ai 3y ago
      Who are they? Why do you mix up the guys who prepared the data with the other guys who used this data and making money from a vague memory of that data?
- idle_zealot 3y ago
  I congratulate all of the authors whose work is included in this dataset on contributing their knowledge, skills, and perspective to humanity's various endeavors, both creative and technical. I hope that the fruits of their labors are returned to them, rather than being selfishly hoarded by the few with the resources necessary to produce those fruits, be they publishers, middlemen, or big tech. Which is all to say that information shouldn't be hoarded and guarded. If it can produce something more than the sum of its parts we should use it to do so. The result of that should, on the same grounds, not be hoarded and guarded, doubly so being based on the work of others.
  - gosub100 3y ago
    It will produce "something more" for the already-wealthy who control the technology. For instance, LLMs will eliminate the need for some customer service jobs, increasing the profit margin for the existing executives and shareholders, while eliminating entry-level jobs from the job market.
    - idle_zealot 3y ago
      Call me an idealist but I don't think humans should be spending their time on jobs a computer can do. The solution to wealth disparity cannot include "invent menial untalented high-paying labor for people to do".
      - __loam 3y ago
        Yeah why should humans do bothersome labor like...creating literature?
        fennecfoxy 3y ago
        You're romanticising the creation of literature for some reason. There's no difference between a machine replacing a human writing a book and a machine replacing a human making a piece of wooden furniture. Literature and craftwork are equally rewarding and beneficial imo. And people still make wooden furniture by hand even after IKEA. If someone wants cheap and uninteresting they'll buy IKEA, if someone wants an interesting and unique piece, or to support handmade things then they'll buy from a woodworker.
jsheard 3y ago
This dataset includes "books3", which is a comprehensive dump of Bibliotik, a torrent tracker dedicated to pirated ebooks. Throw a dart at a wall filled with every notable author/publisher ever and whoever you hit probably owns some of this data. Apparently you can just do whatever as long as you say it's for AI research, go post Blu-ray rips online, it's fine provided you have a .ai domain :^)
- fsckboy 3y ago
  > Throw a dart at a wall filled with every notable author/publisher ever copyrights do expire, and any books older than Mickey Mouse are public domain, so it's not every notable author ever
  - jsheard 3y ago
    Technically true, narrow that down to merely "every notable living author and a subset of dead ones" then. Bram Stokers bones will be relieved to hear that their work isn't being misappropriated.
- oldgradstudent 3y ago
  It also contains an archive of opensubtitles, which is also not very open source.
  - refulgentis 3y ago
    The subtitles aren't open? If you meant transcribing dialogue from a TV show is violating copyright, I'm not so sure, it's relatively common to quote dialogue for varied purposes, ex. TV critics Definitely understand if you're saying the whole dialogue for a TV show is copyrighted, but I'm curious about the opensubtitles part, used to work in that area.
    - PavleMiha 3y ago
      Quoting is very different from posting the full contents of something. I can quote a book but I can’t reproduce it in its entirety.

__loam 3y ago

They stole it because they think building their toys is more important than everyone else's rights to the product of their own labor.

johndough 3y ago
I doubt that anyone is going to download and search through over 800 TB just to find a badly formatted copy of some book that could be found much quicker on different websites with better formatting. Authors are losing fractional cents here at most.
- gosub100 3y ago
  so just like Office Space? (paraphrasing) "We steal a fraction of a cent from each transaction, who do we hurt? Nobody. We just put the remainder into our account!" Sorry that's not how damages are calculated in the US tort system.
  - johndough 3y ago
    I do not know how damages are calculated in the US tort system. What do they say about the books3 dataset? I also think that the case is different here, since in your example, there is a specific amount of money being stolen, while in the books3 case, there is an unspecified amount of money not being made by the authors.
    - SEGyges 3y ago
      I am pretty sure if the authors were trying to license their works for this purpose we would just not use them at all; it is difficult to see under what circumstances they would stand to profit from this other than by suing people after the fact over it.
      - doug_durham 3y ago
        I think you could argue that authors could profit from their works being cited in an LLM response. It could drive sales of their works much like citations do on the web. The counter argument is that and LLM could give you the Clif Notes version of the work and thus taking away a portion of sales.
        SEGyges 3y ago
        In a world where the options were to 1) pay the author, 2) implement guaranteed citation of the author any time the model gave an answer that was directly derivative, with an option to not do so if the summary was sufficiently vague, or 3) ignore the author's book completely as training data we would all choose 3).
        __loam 3y ago
        And the authors would probably be very happy that you did.
- __loam 3y ago
  The penalty is up to $150k per violation.
  - sp332 3y ago
    For uploading, not downloading.
pk-protect-ai 3y ago
they have stole nothing, they make no profit from it as well.
- idle_zealot 3y ago
  Oh, so there aren't AI companies charging for access to private models?
  - pk-protect-ai 3y ago
    Who are they? Why do you mix up the guys who prepared the data with the other guys who used this data and making money from a vague memory of that data?
idle_zealot 3y ago
I congratulate all of the authors whose work is included in this dataset on contributing their knowledge, skills, and perspective to humanity's various endeavors, both creative and technical. I hope that the fruits of their labors are returned to them, rather than being selfishly hoarded by the few with the resources necessary to produce those fruits, be they publishers, middlemen, or big tech. Which is all to say that information shouldn't be hoarded and guarded. If it can produce something more than the sum of its parts we should use it to do so. The result of that should, on the same grounds, not be hoarded and guarded, doubly so being based on the work of others.
- gosub100 3y ago
  It will produce "something more" for the already-wealthy who control the technology. For instance, LLMs will eliminate the need for some customer service jobs, increasing the profit margin for the existing executives and shareholders, while eliminating entry-level jobs from the job market.
  - idle_zealot 3y ago
    Call me an idealist but I don't think humans should be spending their time on jobs a computer can do. The solution to wealth disparity cannot include "invent menial untalented high-paying labor for people to do".
    - __loam 3y ago
      Yeah why should humans do bothersome labor like...creating literature?
      - fennecfoxy 3y ago
        You're romanticising the creation of literature for some reason. There's no difference between a machine replacing a human writing a book and a machine replacing a human making a piece of wooden furniture. Literature and craftwork are equally rewarding and beneficial imo. And people still make wooden furniture by hand even after IKEA. If someone wants cheap and uninteresting they'll buy IKEA, if someone wants an interesting and unique piece, or to support handmade things then they'll buy from a woodworker.

jsheard 3y ago

This dataset includes "books3", which is a comprehensive dump of Bibliotik, a torrent tracker dedicated to pirated ebooks. Throw a dart at a wall filled with every notable author/publisher ever and whoever you hit probably owns some of this data. Apparently you can just do whatever as long as you say it's for AI research, go post Blu-ray rips online, it's fine provided you have a .ai domain :^)

fsckboy 3y ago
> Throw a dart at a wall filled with every notable author/publisher ever copyrights do expire, and any books older than Mickey Mouse are public domain, so it's not every notable author ever
- jsheard 3y ago
  Technically true, narrow that down to merely "every notable living author and a subset of dead ones" then. Bram Stokers bones will be relieved to hear that their work isn't being misappropriated.
oldgradstudent 3y ago
It also contains an archive of opensubtitles, which is also not very open source.
- refulgentis 3y ago
  The subtitles aren't open? If you meant transcribing dialogue from a TV show is violating copyright, I'm not so sure, it's relatively common to quote dialogue for varied purposes, ex. TV critics Definitely understand if you're saying the whole dialogue for a TV show is copyrighted, but I'm curious about the opensubtitles part, used to work in that area.
  - PavleMiha 3y ago
    Quoting is very different from posting the full contents of something. I can quote a book but I can’t reproduce it in its entirety.