6 ms·

It is great to see that they even trained it on open datasets. More AI models need to do this, especially if they market themselves as open.

mdp2021 2y ago
Why? A great part of the important material for knowledge and mandatory intellectual exercise is not "open" (possibly "accessible", yet under copy rights).
- rafram 2y ago
  It’s difficult to argue that a model is truly “open” if the creator won’t even tell you what they trained it on. Even as companies like Meta argue that training on copyrighted material is OK, they still don’t want to openly admit that that’s what they did - and providing their training data, which would likely be a giant list of books from LibGen, would give the game away.
  - perfmode 2y ago
    they stated it in the original llama paper
    - bredren 2y ago
      The 2/27/23 paper? Where?
    - rafram 2y ago
      Right, the original LLaMA paper. LLaMA 2 and 3 are significantly more capable models trained on orders of magnitude more data, and those papers notably do not say where the data comes from. The LLaMA 3 paper helpfully mentions that "Much of the data we utilize is obtained from the web," so I guess that's better than nothing! LLaMA 2: https://arxiv.org/pdf/2307.09288 https://arxiv.org/pdf/2307.09288 LLaMA 3: https://arxiv.org/pdf/2407.21783 https://arxiv.org/pdf/2407.21783
  - mdp2021 2y ago
    > It’s difficult to argue that a model is truly “open” if the creator won’t even tell you what they trained it on But the original poster said being glad «they even trained it on open datasets», not glad "they told you what they trained it on".
    - rafram 2y ago
      I read "open dataset" as "training data URLs are in a CSV somewhere," but I see how you could've read it differently.
  - NewJazz 2y ago
    Yeah, I'm not fundamentally opposed to them training on copyrighted material, though I expect at some point to be thoroughly frustrated while trying to access such a dataset. I do take personal affront when they dump models without specifying their datasets. They're just polluting the information space at that point.
  - Workaccount2 2y ago
    This is a good place to interject a point about copyright training that people overlook, conflate, or don't realize. There are two types of "training on copyright material" 1.) Training on material that is copyrighted and behind a paywall, but you circumvent the paywall. This is unambiguously illegal, as the material is pay to view. 2.)Training on material that is copyrighted, but free for anyone to consume. This is ambiguous in legality, but right now seems to be leaning in AI training favor - as long as the models don't verbatim share the material. There is also another point about copyright material being ad-supported, and obviously the AI doesn't view/care about ads. There is a decent case to be made that this is illegal, but then is ad blocking actually theft?
- esafak 2y ago
  That's not AMD's fault. They are abiding by the law. If IP holders licensed or shared their data, AMD could train on them.
  - t-3 2y ago
    It's important to note that AMD is an IP company, Meta is a data company. AMD would be shooting itself in the foot if it normalized flagrantly violating the IP of others. Meta doesn't care about IP, they just want to sell ads and data.
  - mdp2021 2y ago
    You must have misunderstood the post: the OP wrote «great to see that they even trained it on open datasets», to which I replied "why should that be «great»: machines with sought "intellectual" abilities must be trained on a larger corpus than just open". The point is not about any «AMD's fault». It is about "why would it be great to have LLMs trained on limited (open) data".
ogrisel 2y ago
It appears that they reused a lot of the data preparation provided by the AllenAI team: https://github.com/allenai/OLMoE https://github.com/allenai/OLMoE https://github.com/allenai/dolma https://github.com/allenai/dolma https://github.com/AMD-AIG-AIMA/Instella https://github.com/AMD-AIG-AIMA/Instella

mdp2021 2y ago

Why? A great part of the important material for knowledge and mandatory intellectual exercise is not "open" (possibly "accessible", yet under copy rights).

rafram 2y ago
It’s difficult to argue that a model is truly “open” if the creator won’t even tell you what they trained it on. Even as companies like Meta argue that training on copyrighted material is OK, they still don’t want to openly admit that that’s what they did - and providing their training data, which would likely be a giant list of books from LibGen, would give the game away.
- perfmode 2y ago
  they stated it in the original llama paper
  - bredren 2y ago
    The 2/27/23 paper? Where?
  - rafram 2y ago
    Right, the original LLaMA paper. LLaMA 2 and 3 are significantly more capable models trained on orders of magnitude more data, and those papers notably do not say where the data comes from. The LLaMA 3 paper helpfully mentions that "Much of the data we utilize is obtained from the web," so I guess that's better than nothing! LLaMA 2: https://arxiv.org/pdf/2307.09288 https://arxiv.org/pdf/2307.09288 LLaMA 3: https://arxiv.org/pdf/2407.21783 https://arxiv.org/pdf/2407.21783
- mdp2021 2y ago
  > It’s difficult to argue that a model is truly “open” if the creator won’t even tell you what they trained it on But the original poster said being glad «they even trained it on open datasets», not glad "they told you what they trained it on".
  - rafram 2y ago
    I read "open dataset" as "training data URLs are in a CSV somewhere," but I see how you could've read it differently.
- NewJazz 2y ago
  Yeah, I'm not fundamentally opposed to them training on copyrighted material, though I expect at some point to be thoroughly frustrated while trying to access such a dataset. I do take personal affront when they dump models without specifying their datasets. They're just polluting the information space at that point.
- Workaccount2 2y ago
  This is a good place to interject a point about copyright training that people overlook, conflate, or don't realize. There are two types of "training on copyright material" 1.) Training on material that is copyrighted and behind a paywall, but you circumvent the paywall. This is unambiguously illegal, as the material is pay to view. 2.)Training on material that is copyrighted, but free for anyone to consume. This is ambiguous in legality, but right now seems to be leaning in AI training favor - as long as the models don't verbatim share the material. There is also another point about copyright material being ad-supported, and obviously the AI doesn't view/care about ads. There is a decent case to be made that this is illegal, but then is ad blocking actually theft?
esafak 2y ago
That's not AMD's fault. They are abiding by the law. If IP holders licensed or shared their data, AMD could train on them.
- t-3 2y ago
  It's important to note that AMD is an IP company, Meta is a data company. AMD would be shooting itself in the foot if it normalized flagrantly violating the IP of others. Meta doesn't care about IP, they just want to sell ads and data.
- mdp2021 2y ago
  You must have misunderstood the post: the OP wrote «great to see that they even trained it on open datasets», to which I replied "why should that be «great»: machines with sought "intellectual" abilities must be trained on a larger corpus than just open". The point is not about any «AMD's fault». It is about "why would it be great to have LLMs trained on limited (open) data".

ogrisel 2y ago

It appears that they reused a lot of the data preparation provided by the AllenAI team: https://github.com/allenai/OLMoE https://github.com/allenai/OLMoE https://github.com/allenai/dolma https://github.com/allenai/dolma https://github.com/AMD-AIG-AIMA/Instella https://github.com/AMD-AIG-AIMA/Instella