4 ms·

Or books3. :) (I put together a dataset of 190,000 books that I called books3, which llama eventually trained on. Usage rights are a big interest of mine, prim

Or books3. :) (I put together a dataset of 190,000 books that I called books3, which llama eventually trained on. Usage rights are a big interest of mine, primarily because there’s a weird disconnect of people trying to claim copyright over models when the underlying data obviously wasn’t copyrightable.)

selestify 3y ago
A model (if trained right) will not include the training texts verbatim, and will arguably be a big enough transformation over the source material to be copyrightable in its own right. IANAL, and I’m not saying how this is how it’ll play out in the courts, but I don’t see why this is a “weird disconnect”
- _ea1k 3y ago
  I suspect the challenge is avoiding cases where answers are overfit. Can you get it to give you the full text of a chapter in a book that it has read, for example? It'd be easy to imagine models that accidentally made this possible.
yyyk 3y ago
Thanks for your work! I had already downloaded the entire dataset. But I wonder what was the criteria behind book selection? I found it difficult to understand at places*. * Sometimes a book has several very slightly different versions. A lot of Fantasy. Sometimes many parts of a series are included but not all of it (I can imagine an argument for a sample, or for the series in its entirity). Commentaries about important philosophical arguments like Rawls' 'A theory of justice', but not the work itself.
- sillysaurusx 3y ago
  It was bibliotik. Embarrassingly I didn’t even think that duplicate books could be a problem. The llama folks had to dedupe the books themselves. Someday I’d like to do that too and release a cleaned version. Basically, the-eye.eu was at one point hosting all of bibliotik, so I downloaded all the epubs and converted them to text. I still have those epubs (incidentally thanks to Carmack, who through a convoluted process managed to save the them and send them to me via snail mail) and I’ve been considering releasing them so that you can filter the books yourself.

selestify 3y ago

A model (if trained right) will not include the training texts verbatim, and will arguably be a big enough transformation over the source material to be copyrightable in its own right. IANAL, and I’m not saying how this is how it’ll play out in the courts, but I don’t see why this is a “weird disconnect”

_ea1k 3y ago
I suspect the challenge is avoiding cases where answers are overfit. Can you get it to give you the full text of a chapter in a book that it has read, for example? It'd be easy to imagine models that accidentally made this possible.

yyyk 3y ago

Thanks for your work! I had already downloaded the entire dataset. But I wonder what was the criteria behind book selection? I found it difficult to understand at places*. * Sometimes a book has several very slightly different versions. A lot of Fantasy. Sometimes many parts of a series are included but not all of it (I can imagine an argument for a sample, or for the series in its entirity). Commentaries about important philosophical arguments like Rawls' 'A theory of justice', but not the work itself.

sillysaurusx 3y ago
It was bibliotik. Embarrassingly I didn’t even think that duplicate books could be a problem. The llama folks had to dedupe the books themselves. Someday I’d like to do that too and release a cleaned version. Basically, the-eye.eu was at one point hosting all of bibliotik, so I downloaded all the epubs and converted them to text. I still have those epubs (incidentally thanks to Carmack, who through a convoluted process managed to save the them and send them to me via snail mail) and I’ve been considering releasing them so that you can filter the books yourself.