4 ms·

It’s difficult to argue that a model is truly “open” if the creator won’t even tell you what they trained it on. Even as companies like Meta argue that training

by rafram 2y ago

It’s difficult to argue that a model is truly “open” if the creator won’t even tell you what they trained it on. Even as companies like Meta argue that training on copyrighted material is OK, they still don’t want to openly admit that that’s what they did - and providing their training data, which would likely be a giant list of books from LibGen, would give the game away.

perfmode 2y ago
they stated it in the original llama paper
- bredren 2y ago
  The 2/27/23 paper? Where?
- rafram 2y ago
  Right, the original LLaMA paper. LLaMA 2 and 3 are significantly more capable models trained on orders of magnitude more data, and those papers notably do not say where the data comes from. The LLaMA 3 paper helpfully mentions that "Much of the data we utilize is obtained from the web," so I guess that's better than nothing! LLaMA 2: https://arxiv.org/pdf/2307.09288 https://arxiv.org/pdf/2307.09288 LLaMA 3: https://arxiv.org/pdf/2407.21783 https://arxiv.org/pdf/2407.21783
mdp2021 2y ago
> It’s difficult to argue that a model is truly “open” if the creator won’t even tell you what they trained it on But the original poster said being glad «they even trained it on open datasets», not glad "they told you what they trained it on".
- rafram 2y ago
  I read "open dataset" as "training data URLs are in a CSV somewhere," but I see how you could've read it differently.
NewJazz 2y ago
Yeah, I'm not fundamentally opposed to them training on copyrighted material, though I expect at some point to be thoroughly frustrated while trying to access such a dataset. I do take personal affront when they dump models without specifying their datasets. They're just polluting the information space at that point.
Workaccount2 2y ago
This is a good place to interject a point about copyright training that people overlook, conflate, or don't realize. There are two types of "training on copyright material" 1.) Training on material that is copyrighted and behind a paywall, but you circumvent the paywall. This is unambiguously illegal, as the material is pay to view. 2.)Training on material that is copyrighted, but free for anyone to consume. This is ambiguous in legality, but right now seems to be leaning in AI training favor - as long as the models don't verbatim share the material. There is also another point about copyright material being ad-supported, and obviously the AI doesn't view/care about ads. There is a decent case to be made that this is illegal, but then is ad blocking actually theft?

by rafram 2y ago

perfmode 2y ago

they stated it in the original llama paper

bredren 2y ago
The 2/27/23 paper? Where?
rafram 2y ago
Right, the original LLaMA paper. LLaMA 2 and 3 are significantly more capable models trained on orders of magnitude more data, and those papers notably do not say where the data comes from. The LLaMA 3 paper helpfully mentions that "Much of the data we utilize is obtained from the web," so I guess that's better than nothing! LLaMA 2: https://arxiv.org/pdf/2307.09288 https://arxiv.org/pdf/2307.09288 LLaMA 3: https://arxiv.org/pdf/2407.21783 https://arxiv.org/pdf/2407.21783

mdp2021 2y ago

> It’s difficult to argue that a model is truly “open” if the creator won’t even tell you what they trained it on But the original poster said being glad «they even trained it on open datasets», not glad "they told you what they trained it on".

rafram 2y ago
I read "open dataset" as "training data URLs are in a CSV somewhere," but I see how you could've read it differently.

NewJazz 2y ago

Yeah, I'm not fundamentally opposed to them training on copyrighted material, though I expect at some point to be thoroughly frustrated while trying to access such a dataset. I do take personal affront when they dump models without specifying their datasets. They're just polluting the information space at that point.

Workaccount2 2y ago

This is a good place to interject a point about copyright training that people overlook, conflate, or don't realize. There are two types of "training on copyright material" 1.) Training on material that is copyrighted and behind a paywall, but you circumvent the paywall. This is unambiguously illegal, as the material is pay to view. 2.)Training on material that is copyrighted, but free for anyone to consume. This is ambiguous in legality, but right now seems to be leaning in AI training favor - as long as the models don't verbatim share the material. There is also another point about copyright material being ad-supported, and obviously the AI doesn't view/care about ads. There is a decent case to be made that this is illegal, but then is ad blocking actually theft?