5 ms·
Are we all reading the same complaint? They say: > in a Meta paper detailing LLaMA, the company points to sources for its training datasets, one of which is
by edent 3y ago
Are we all reading the same complaint?
They say:
> in a Meta paper detailing LLaMA, the company points to sources for its training datasets, one of which is called ThePile, which was assembled by a company called EleutherAI. ThePile, the complaint points out, was described in an EleutherAI paper as being put together from “a copy of the contents of the Bibliotik private tracker.”
Does that stack up?
The Meta Paper - https://arxiv.org/pdf/2302.13971.pdf https://arxiv.org/pdf/2302.13971.pdf - says:
> We include two book corpora in our training dataset: the Gutenberg Project, which contains books that are in the public domain, and the Books3 section of ThePile (Gao et al., 2020)
The Pile Paper - https://arxiv.org/abs/2101.00027 https://arxiv.org/abs/2101.00027 - says it was trained (in part) on "Books3" which it describes as:
> Books3 is a dataset of books derived from a copy of the contents of the Bibliotik private tracker made available by Shawn Presser (Presser, 2020).
Shawn Presser's link is at https://twitter.com/theshawwn/status/1320282149329784833 https://twitter.com/theshawwn/status/1320282149329784833 and he describes Book3 as
> Presenting "books3", aka "all of bibliotik" - 196,640 books - in plain .txt
I don't have the time and space to download the 37GB file. But if Silverman's book is in there... isn't this a slam dunk case?
Meta's LLaMA is - as they seem to admit - trained on pirated books.
- moffkalast 3y agoWe don't seem to be reading the same thing, you're pulling Google out of thin air somewhere.
- edent 3y agoI'm literally quoting The Verge article and following the links they present...
- moffkalast 3y agoMust be quoting it wrong then, Google has nothing to do with LLama.
- edent 3y agoSorry, it's late here. I meant Meta. Thanks for the correction.
- amf12 3y agoThat paper is by Meta AI. Where are you getting Google from?
- pessimizer 3y agoYou might as well be complaining about the grammar. This is what was said in the article. > The complaint lays out in steps why the plaintiffs believe the datasets have illicit origins — in a Meta paper detailing LLaMA, the company points to sources for its training datasets, one of which is called ThePile, which was assembled by a company called EleutherAI. ThePile, the complaint points out, was described in an EleutherAI paper as being put together from “a copy of the contents of the Bibliotik private tracker.” Bibliotik and the other “shadow libraries” listed, says the lawsuit, are “flagrantly illegal.”
- banana_giraffe 3y ago> But if Silverman's book is in there It is: $ grep -i "Sarah Silverman" books3.list.txt 325196 books3/the-eye.eu/public/Books/Bibliotik/T/The Bedwetter - Sarah Silverman.epub.txt Anyone that just wants to see the list of files (itself a big file): https://gist.githubusercontent.com/Q726kbXuN/e4e9919a2f5d81f1c2b23bad53dda4d1/raw/99ba9cf49e7a75c7c0f9de58ab6e7bd94bf196ca/books3.list.txt https://gist.githubusercontent.com/Q726kbXuN/e4e9919a2f5d81f...
- dragonwriter 3y ago> I don't have the time and space to download the 37GB file. But if Silverman's book is in there... isn't this a slam dunk case? Yes, and no. Its pretty much a slam dunk case that, insofar as the initial training required causing a copy of the corpus defined by the tracker to be made as part of the process, it involved an act violating copyright. Whether that entitles Silverman to any remedy beyond compensation for (maybe treble damages for) the equivalent of the purchase price of the book depends on... well, basically the same issues of how copyright relates to model training (and an additional argument about whether the illicit status of the material before the training modifies that).