4 ms·

IANAL, but this seems very fishy to me: 1) I don't understand how this isn't a derivative work of the original code, as I very highly doubt you've done a clea

by querez 4y ago

IANAL, but this seems very fishy to me: 1) I don't understand how this isn't a derivative work of the original code, as I very highly doubt you've done a clean room implementation. I doubt this would hold up in court. 2) Doesn't the original FB license also apply to the weights? Just re-implementing the code would not change the license on the weights. So while THE CODE may now be re-licensed, the weights would still fall under the original license. I'd love if someone with more legal understanding could shed some light on this.

MacsHeadroom 4y ago
>I don't understand how this isn't a derivative work of the original code The original code is Apache 2 licensed. Derivatives are fine and allowed. This retains the same Apache 2 license as Facebook's code. It's only the model that isn't covered by that permissive Apache 2 license. A model produced by a derivative of the permissively licensed code, or even by the original code itself, is not a derivative or the original non-permissively licensed model produced by the original code and is non-infringing even if it is a bit-perfect replica. > Doesn't the original FB license also apply to the weights? Again, there are different licenses for the code and the model and neither license actually applies to the weights within the model only the actual exact model. If this project produced a bit-for-bit replica of Facebook's model it would still not infringe on that model's license. But it doesn't produce a bit-for-bit replica. Even if Facebook were to re-run their same training code on their same hardware would they could not produce the exact same weights as before since massively parallel matrix multiplications are not deterministic. Benign environmental noise like microscopic fluctuations in temperature make a difference in the outcome.
- kristjansson 4y ago
  > Apache 2 Isn't the original GPLv3[0]? [0]: https://github.com/facebookresearch/llama/blob/main/LICENSE https://github.com/facebookresearch/llama/blob/main/LICENSE
  - lantiga 4y ago
    Correct, the original is GPL 3. To produce this implementation from the LLaMA paper we started from github.com/karpathy/nanoGPT, the LLaMA architecture is really similar to GPT. For instance we added rotary positional encoding starting from the original RoPE repo published with the paper. We finally ran the original model to make sure the two models were numerically.
- shagie 4y ago
  Can a model itself be copyrighted? https://www.copyright.gov/circs/circ33.pdf https://www.copyright.gov/circs/circ33.pdf > To register a work with the U.S. Copyright Office, you must identify the copyrightable subject matter forming the basis of your claim. To be copyrightable, a work must qualify as an original work of authorship, meaning that it must have been created independently and contain a sufficient amount of creativity. Most works meet these conditions. Some works, however, contain elements that either lack the required creativity or are placed outside the bounds of copyright by the law. This circular highlights different types of noncopyrightable subject matter. For more information, see chapter 300, section 313.3, of the Compendium of U.S. Copyright Office Practices.1 https://www.copyright.gov/comp3/chap300/ch300-copyrightable-authorship.pdf https://www.copyright.gov/comp3/chap300/ch300-copyrightable-... > 313.2 Works That Lack Human Authorship > Similarly, the Office will not register works produced by a machine or mere mechanical process that operates randomly or automatically without any creative input or intervention from a human author. The crucial question is “whether the ‘work’ is basically one of human authorship, with the computer [or other device] merely being an assisting instrument, or whether the traditional elements of authorship in the work (literary, artistic, or musical expression or elements of selection, arrangement, etc.) were actually conceived and executed not by man but by a machine.” > Examples > Reducing or enlarging the size of a preexisting work of authorship. ---- I'm going to go with that models themselves lack human authorship or any sufficient amount of creativity and thus aren't copyrightable. They may fall under trade secret... but they need to be treated as one. This would also mean that licenses based on copyright (most of FOSS licenses) wouldn't be applicable to them.
rnosov 4y ago
1) I've looked at both codebases and this one is definitely a derivative of the nanoGPT. You can compare all three implementations yourself as they are actually surprisingly compact and readable. 2) The issue whether weights are copyrightable at all has not been settled yet. If they are, there is a fair use doctrine that allows transformative works of a copyrighted work. The line is a bit blurry but consider Cariou v. Prince case[1] where addition of colour to some black and white photos was considered enough to be transformative. Similarly, full fine tuning on current news or adding visual modality could potentially create a brand new model in the eyes of a law. [1] https://cyber.harvard.edu/people/tfisher/cx/2013_Cariou.pdf https://cyber.harvard.edu/people/tfisher/cx/2013_Cariou.pdf
mostdataisnice 4y ago
This is the clearest example of an attention grab I have seen - it does nothing for commercial use of Llama unless they provide a version of the weights produced by them and not Facebook. (and they don't...they ask you to download them from Facebook's repo)

by querez 4y ago

MacsHeadroom 4y ago

>I don't understand how this isn't a derivative work of the original code The original code is Apache 2 licensed. Derivatives are fine and allowed. This retains the same Apache 2 license as Facebook's code. It's only the model that isn't covered by that permissive Apache 2 license. A model produced by a derivative of the permissively licensed code, or even by the original code itself, is not a derivative or the original non-permissively licensed model produced by the original code and is non-infringing even if it is a bit-perfect replica. > Doesn't the original FB license also apply to the weights? Again, there are different licenses for the code and the model and neither license actually applies to the weights within the model only the actual exact model. If this project produced a bit-for-bit replica of Facebook's model it would still not infringe on that model's license. But it doesn't produce a bit-for-bit replica. Even if Facebook were to re-run their same training code on their same hardware would they could not produce the exact same weights as before since massively parallel matrix multiplications are not deterministic. Benign environmental noise like microscopic fluctuations in temperature make a difference in the outcome.

kristjansson 4y ago
> Apache 2 Isn't the original GPLv3[0]? [0]: https://github.com/facebookresearch/llama/blob/main/LICENSE https://github.com/facebookresearch/llama/blob/main/LICENSE
- lantiga 4y ago
  Correct, the original is GPL 3. To produce this implementation from the LLaMA paper we started from github.com/karpathy/nanoGPT, the LLaMA architecture is really similar to GPT. For instance we added rotary positional encoding starting from the original RoPE repo published with the paper. We finally ran the original model to make sure the two models were numerically.
shagie 4y ago
Can a model itself be copyrighted? https://www.copyright.gov/circs/circ33.pdf https://www.copyright.gov/circs/circ33.pdf > To register a work with the U.S. Copyright Office, you must identify the copyrightable subject matter forming the basis of your claim. To be copyrightable, a work must qualify as an original work of authorship, meaning that it must have been created independently and contain a sufficient amount of creativity. Most works meet these conditions. Some works, however, contain elements that either lack the required creativity or are placed outside the bounds of copyright by the law. This circular highlights different types of noncopyrightable subject matter. For more information, see chapter 300, section 313.3, of the Compendium of U.S. Copyright Office Practices.1 https://www.copyright.gov/comp3/chap300/ch300-copyrightable-authorship.pdf https://www.copyright.gov/comp3/chap300/ch300-copyrightable-... > 313.2 Works That Lack Human Authorship > Similarly, the Office will not register works produced by a machine or mere mechanical process that operates randomly or automatically without any creative input or intervention from a human author. The crucial question is “whether the ‘work’ is basically one of human authorship, with the computer [or other device] merely being an assisting instrument, or whether the traditional elements of authorship in the work (literary, artistic, or musical expression or elements of selection, arrangement, etc.) were actually conceived and executed not by man but by a machine.” > Examples > Reducing or enlarging the size of a preexisting work of authorship. ---- I'm going to go with that models themselves lack human authorship or any sufficient amount of creativity and thus aren't copyrightable. They may fall under trade secret... but they need to be treated as one. This would also mean that licenses based on copyright (most of FOSS licenses) wouldn't be applicable to them.

rnosov 4y ago

1) I've looked at both codebases and this one is definitely a derivative of the nanoGPT. You can compare all three implementations yourself as they are actually surprisingly compact and readable. 2) The issue whether weights are copyrightable at all has not been settled yet. If they are, there is a fair use doctrine that allows transformative works of a copyrighted work. The line is a bit blurry but consider Cariou v. Prince case[1] where addition of colour to some black and white photos was considered enough to be transformative. Similarly, full fine tuning on current news or adding visual modality could potentially create a brand new model in the eyes of a law. [1] https://cyber.harvard.edu/people/tfisher/cx/2013_Cariou.pdf https://cyber.harvard.edu/people/tfisher/cx/2013_Cariou.pdf

mostdataisnice 4y ago

This is the clearest example of an attention grab I have seen - it does nothing for commercial use of Llama unless they provide a version of the weights produced by them and not Facebook. (and they don't...they ask you to download them from Facebook's repo)