3 ms·
The models that are trained on these inputs absolutely do not contain all of the text that they're trained on. Even with non-quantized models you're looking at
by rpdillon 3y ago
The models that are trained on these inputs absolutely do not contain all of the text that they're trained on. Even with non-quantized models you're looking at somewhere in the realm of 3% of the training data being present in the final model.
Further, these models are not used to reproduce copyrighted works... they would be one of the least efficient ways to do so.
I also don't think there's a particularly strong argument that training on a book or image decreases the market for the original book or image. To take an example, zero folks that want to read Game of Thrones are asking ChatGPT to generate Game of Thrones for them instead of buying the book.
I think what's really happened here is that artists' work has utility outside of the originally intended audience, that utility largely comes from statistically analyzing the writing, and the debate is about whether or not the utility of those statistics requires additional compensation for the artists. That question seems to hinge on the intricacies of copyright law, but as I've stated before, I think the fair use argument is quite strong.
- pbhjpbhj 3y ago>I also don't think ... that training on a book or image decreases the market for the original book or image. ... zero folks that want to read Game of Thrones are asking ChatGPT to generate Game of Thrones ... For now. But when you can generate work in the style of an artist without paying that artist, I would. More importantly, media production companies will and so will use access to people's post work to put them out of work now. We need to distribute the benefits, we're drifting towards copyright infringement being allowed by rich corporation (with resources to generate "AI") as long as they abstract it through a transformer, or whatever.
- antifa 3y agoCrypto was a decentralized ponzi scheme. AI will be decentralized mass copyright infringement.
- artninja1988 3y agoCopyright laws are fascist and should be abolished
- Turing_Machine 3y ago> But when you can generate work in the style of an artist without paying that artist, I would. You can't copyright a "style".
- joaogui1 3y agoWhere does this 3% figure come from?
- Zuiii 3y agoThese models are generally trained on tarabytes of data, but are usually 10s of gigabytes large (or much less if quantized). The latest true open source model, mistral 7b, is only 3GB (0.3% of a TB) when quantized.
- rpdillon 3y agoI did a very similar analysis with Llama 65B being trained on 5.6T tokens assuming token length of 4 characters and comparing with a quantized model size of ~38GB. The 3% number was a conservative rounding of the same calculation, but retaining fp16 rather than quantizing to 4 bits. Here's my original back of the napkin analysis: https://news.ycombinator.com/item?id=36681440 https://news.ycombinator.com/item?id=36681440
- pclmulqdq 3y ago> To take an example, zero folks that want to read Game of Thrones are asking ChatGPT to generate Game of Thrones for them instead of buying the book. Why jump straight to one of the longest popular books in English? I'm sure plenty of people who would otherwise buy poetry anthologies are asking ChatGPT to write them a haiku. It's also entirely conceivable that ChatGPT can perfectly reproduce a poem in its training set - it can produce paragraph-sized quotes.
- rpdillon 3y agoIt was a reference to the copyright suit brought by the author. https://apnews.com/article/openai-lawsuit-authors-grisham-george-rr-martin-37f9073ab67ab25b7e6b2975b2a63bfe https://apnews.com/article/openai-lawsuit-authors-grisham-ge...