4 ms·

Definition for Open Weights Licensing

39 points by andy99 3y ago

guy98238710 3y ago
Dunno who these guys are, but generally w.r.t. ML licensing, research kit is the important thing nobody is publishing. You know, the code used for diagnostics, visualization, benchmarks, decision support, etc. Research kit can be orders of magnitude larger than the final implementation code. Weights are just a binary with expensive build process.
- andy99 3y ago
  Not just a binary because a common use is to fine tune from them to create derivative works. Look at all the models based on LLaMA.
  - orra 3y ago
    Yes, precisely. If fine tuning is the preferred method of modification, then under the GPL the weights are source code. Personally I think the truth is greyer than black and white.
  - guy98238710 3y ago
    Fine-tuning is like statically linking a large library (available in binary form only) to a small app. (An optimizing linker can blend app and library code so much they cannot be distinguished.) Weights intended for fine-tuning are still a binary, just a reusable one. Nothing new here.
fullstackchris 3y ago
This is a great idea, but I'm thinking weights aren't enough. If the recent rumors are true of the latest models like chat GPT-4, for example - they have a realitively new style of overlapping _a cohort_ of models, not just utilizing a giant matrix of weights (as has hereforto been the defacto architecture) - or is this still the case even for chat gpt 4 after training? I'm no expert in ML, so perhaps I am ignorant as to what information is truly needed to reproduce results. I just cant help but think you'd need some hints at the model architecture itself as well.
- famouswaffles 3y ago
  There's nothing new about a Mixture of Experts architecture( and it's still essentially a giant matrix of weights). They just consistently well underperformed their dense counterparts so people were surprised. But this recent paper shows MoE seem to benefit a lot better with Instruct training than their dense counterparts. https://arxiv.org/abs/2305.14705 https://arxiv.org/abs/2305.14705
mellosouls 3y ago
Discussion from the other day on a similarly themed article. https://news.ycombinator.com/item?id=36601622 https://news.ycombinator.com/item?id=36601622
naillo 3y ago
If someone wants to release weights but attach a no commercial license to it I'm somewhat fine with that. I won't be able to sell it behind a shallow API but I'll be able to use the output of the model (that can't be prohibited) and sell that or do whatever I like with it. It's not as ideal as CC0 but I'm ok with the compromise.
- orra 3y ago
  > I'll be able to use the output of the model (that can't be prohibited) If the model is copyrightable, they can prohibit you from running the model for a commercial purpose, even if they don't have copyright on the output.
  - ivalm 3y ago
    Not sure why you are downvoted, this is exactly my concerns with llama. You can’t run llama with the intent of distilling your own commercial model.
    - orra 3y ago
      (Maybe it's wishful thinking. But, precisely. My analogy would be: you can't run Visual C++ for commercial purposes without a valid licence, even though (ignoring linking against the runtime), the license of a compiler doesn't affect the licence of the output. If weights are copyrightable, then you need permission to copy them into computer memory/run them, just like any application.
zer8k 3y ago
This is one of the few places I would actually prefer the GPL. If someone contributes to this repository with it's current license a company can simply get the weights for free and sell them. You've just made it easier to create another OpenAI, not keep the weights open and auditable. To me, the openness and auditability of weights is the most important. Right now we're heading quickly towards a world where CyberDyne dominates the AI landscape and you're either going to be with them or not involved. In that case, forcing a potential user to contribute weights back will create a more robust ecosystem for so-called personal AI. There is no better license for that then some version of GPL.
- andy99 3y ago
  Yes I agree - I actually wrote something about this the other day. There's a need to think about how and when GPL style licensing can be applied to weights, and what terms make sense to accomplish the "software freedom" goals of the license while still keeping it usable. I'm not a lawyer, I've only considered it from the developer angle: http://marble.onl/posts/motivations_for_open_source.html http://marble.onl/posts/motivations_for_open_source.html
- paddw 3y ago
  > a company can simply get the weights for free and sell them. This is the flaw in your logic. If the weights are freely available, they can not be sold for much value. That will in turn mean there is no "another OpenAI", because the core product, AI, will basically be freely available.
  - Tenoke 3y ago
    Having the weights on their own doesn't mean anyone can have their own OpenAI any more than having a core Google index snapshot would mean anyone can spin their own Google.
    - paddw 3y ago
      The timescale difference between the need for weights taking account of recent events and Google searches is huge.
wrs 3y ago
As mentioned in the other thread, there’s not currently a consensus on whether copyright applies to the compiled weights in the first place. If not, there would be nothing to license. It’s quite possible the only way to protect the weights is as a trade secret: hidden behind an API, or with a confidentiality clause as part of a service agreement or the like.

39 points by andy99 3y ago

Dunno who these guys are, but generally w.r.t. ML licensing, research kit is the important thing nobody is publishing. You know, the code used for diagnostics, visualization, benchmarks, decision support, etc. Research kit can be orders of magnitude larger than the final implementation code. Weights are just a binary with expensive build process.

andy99 3y ago
Not just a binary because a common use is to fine tune from them to create derivative works. Look at all the models based on LLaMA.
- orra 3y ago
  Yes, precisely. If fine tuning is the preferred method of modification, then under the GPL the weights are source code. Personally I think the truth is greyer than black and white.
- guy98238710 3y ago
  Fine-tuning is like statically linking a large library (available in binary form only) to a small app. (An optimizing linker can blend app and library code so much they cannot be distinguished.) Weights intended for fine-tuning are still a binary, just a reusable one. Nothing new here.

fullstackchris 3y ago

This is a great idea, but I'm thinking weights aren't enough. If the recent rumors are true of the latest models like chat GPT-4, for example - they have a realitively new style of overlapping _a cohort_ of models, not just utilizing a giant matrix of weights (as has hereforto been the defacto architecture) - or is this still the case even for chat gpt 4 after training? I'm no expert in ML, so perhaps I am ignorant as to what information is truly needed to reproduce results. I just cant help but think you'd need some hints at the model architecture itself as well.

famouswaffles 3y ago
There's nothing new about a Mixture of Experts architecture( and it's still essentially a giant matrix of weights). They just consistently well underperformed their dense counterparts so people were surprised. But this recent paper shows MoE seem to benefit a lot better with Instruct training than their dense counterparts. https://arxiv.org/abs/2305.14705 https://arxiv.org/abs/2305.14705

mellosouls 3y ago

Discussion from the other day on a similarly themed article. https://news.ycombinator.com/item?id=36601622 https://news.ycombinator.com/item?id=36601622

naillo 3y ago

If someone wants to release weights but attach a no commercial license to it I'm somewhat fine with that. I won't be able to sell it behind a shallow API but I'll be able to use the output of the model (that can't be prohibited) and sell that or do whatever I like with it. It's not as ideal as CC0 but I'm ok with the compromise.

orra 3y ago
> I'll be able to use the output of the model (that can't be prohibited) If the model is copyrightable, they can prohibit you from running the model for a commercial purpose, even if they don't have copyright on the output.
- ivalm 3y ago
  Not sure why you are downvoted, this is exactly my concerns with llama. You can’t run llama with the intent of distilling your own commercial model.
  - orra 3y ago
    (Maybe it's wishful thinking. But, precisely. My analogy would be: you can't run Visual C++ for commercial purposes without a valid licence, even though (ignoring linking against the runtime), the license of a compiler doesn't affect the licence of the output. If weights are copyrightable, then you need permission to copy them into computer memory/run them, just like any application.

zer8k 3y ago

This is one of the few places I would actually prefer the GPL. If someone contributes to this repository with it's current license a company can simply get the weights for free and sell them. You've just made it easier to create another OpenAI, not keep the weights open and auditable. To me, the openness and auditability of weights is the most important. Right now we're heading quickly towards a world where CyberDyne dominates the AI landscape and you're either going to be with them or not involved. In that case, forcing a potential user to contribute weights back will create a more robust ecosystem for so-called personal AI. There is no better license for that then some version of GPL.

andy99 3y ago
Yes I agree - I actually wrote something about this the other day. There's a need to think about how and when GPL style licensing can be applied to weights, and what terms make sense to accomplish the "software freedom" goals of the license while still keeping it usable. I'm not a lawyer, I've only considered it from the developer angle: http://marble.onl/posts/motivations_for_open_source.html http://marble.onl/posts/motivations_for_open_source.html
paddw 3y ago
> a company can simply get the weights for free and sell them. This is the flaw in your logic. If the weights are freely available, they can not be sold for much value. That will in turn mean there is no "another OpenAI", because the core product, AI, will basically be freely available.
- Tenoke 3y ago
  Having the weights on their own doesn't mean anyone can have their own OpenAI any more than having a core Google index snapshot would mean anyone can spin their own Google.
  - paddw 3y ago
    The timescale difference between the need for weights taking account of recent events and Google searches is huge.

wrs 3y ago

As mentioned in the other thread, there’s not currently a consensus on whether copyright applies to the compiled weights in the first place. If not, there would be nothing to license. It’s quite possible the only way to protect the weights is as a trade secret: hidden behind an API, or with a confidentiality clause as part of a service agreement or the like.