4 ms·

Many clearly think of the weights as a source-like artifact. Also, the model's architecture is sufficiently documented, & supported by open code, for others to

by gojomo 3y ago

Many clearly think of the weights as a source-like artifact. Also, the model's architecture is sufficiently documented, & supported by open code, for others to fine-tune it & run it for generation, which to me justifies describing the model as "open", even if the entire process for creating a new set of high-quality weights isn't. (That's in contrast to "Open"AI's GPT4, about which many architectural details are undisclosed.) Note also: as I understand it, there's so much paralellism, algorithmic randomization, & even floating-point-implementation-instability in how such models are trained that even having the exact same training corpus wouldn't be enough to ensure the same final weights. That would require both Facebook & reproducers to do every calculation, in every stage of prepration, in identical order on identical hardware - a constraint that'd make typical parallel/distributed training optimizations (subject to all sorts of CPU/IO/network jitter) impossible.

simonw 3y ago
Yes it's "open", but if it doesn't fit the OSI definition it's rude to call it "open source".
- gojomo 3y ago
  While I like the Open Source Initiative's early & principled stake-in-the-ground, as a matter of usage, many things get casually called 'open source' that don't fully fit the OSI 'Open Source Definition'. And, with regard to the Llama models, it seems to me that all the actual computer-language "source code" to run & train them is available. The specific objection of the grandparent post, with regard to the non-availability of the full training-data document corpus of non-source-code-text, isn't clearly in violation of the OSI's 10-point definition. There is of course a different problem, a part of the Llama's licensing that does clearly violate the OSI Open Source Definition: its "Additional Commericial Terms" preventing just Meta's biggest competitors from using it – discrimination against persons or groups. But that's not the objection raised above.
- JohnKemeny 3y ago
  "Rude" is a strangely chosen word here. Rude towards whom? I could maybe accept "misleading", but as a person who don't believe OSI has an ownership of the term "open source", I don't think it's misleading either.
  - simonw 3y ago
    It's rude towards the open source community. I used the word "rude" carefully here, because I'm not confident that I can make a case for it being illegal or even necessarily misleading (though I personally think it is) - but I'm happy to declare it "rude", partly because rudeness is in the eye of the beholder so I get to determine if I think something is rude or not myself.
luke-stanley 3y ago
I suspect a lot of people like me are feeling quite a bit "nerd sniped" while trying to read the paper. I am just suggesting open-weights are more fit term than the open source term I saw when trying to read the paper, I was not talking about EXACT reproducible builds, which I agree are probably a stretch with current tech (but obviously not impossible with new tech but possibly slower and not obviously needed anyway). I also know about OpenLLaMA and others but that's not Meta's work, and it's not the work in the paper. I'm pretty sure a lot of the core principles Llama uses are drawn from OpenAI's published research but I agree that there is a lack of openness recently, e.g people having to speculate about even core things like how they presumably use mixture of experts, but "open source" is not the right term for them just because OpenAI is not being open right now. I agree mostly that the model itself is somewhat open but I really think we should consider "open weights" as a much better term, I think it's much more reasonable than saying open source.
espadrine 3y ago
> Many clearly think of the weights as a source-like artifact. I believe releasing the weights is necessary for an open-source AI project, but don’t consider the weights to be source. Indeed, open source is not just about the code. For instance, the OSI definition[0] states: > The license must allow modifications and derived works If a company released source code for a project, but it was written in a new language whose compiler they held private, then making derived works would be extremely hard. Someone would have to reverse engineer how the language works from source examples, and reimplement a compiler with a similar performance. In that case, in my view, while the code was open, the project would not be open-source. The same is true of AI models: people can certainly reverse-engineer the training code and replicate a training run to get the weights, but without the weights, the project is not open-source. On the flip side, if the inference code is open and has the weights public but the training code is not, it is similar to an open-source project using a proprietary compiler (think C# before 2014). The project is open-source, but the training is not. [0]: https://opensource.org/osd/ https://opensource.org/osd/

by gojomo 3y ago

simonw 3y ago

Yes it's "open", but if it doesn't fit the OSI definition it's rude to call it "open source".

gojomo 3y ago
While I like the Open Source Initiative's early & principled stake-in-the-ground, as a matter of usage, many things get casually called 'open source' that don't fully fit the OSI 'Open Source Definition'. And, with regard to the Llama models, it seems to me that all the actual computer-language "source code" to run & train them is available. The specific objection of the grandparent post, with regard to the non-availability of the full training-data document corpus of non-source-code-text, isn't clearly in violation of the OSI's 10-point definition. There is of course a different problem, a part of the Llama's licensing that does clearly violate the OSI Open Source Definition: its "Additional Commericial Terms" preventing just Meta's biggest competitors from using it – discrimination against persons or groups. But that's not the objection raised above.
JohnKemeny 3y ago
"Rude" is a strangely chosen word here. Rude towards whom? I could maybe accept "misleading", but as a person who don't believe OSI has an ownership of the term "open source", I don't think it's misleading either.
- simonw 3y ago
  It's rude towards the open source community. I used the word "rude" carefully here, because I'm not confident that I can make a case for it being illegal or even necessarily misleading (though I personally think it is) - but I'm happy to declare it "rude", partly because rudeness is in the eye of the beholder so I get to determine if I think something is rude or not myself.

luke-stanley 3y ago

I suspect a lot of people like me are feeling quite a bit "nerd sniped" while trying to read the paper. I am just suggesting open-weights are more fit term than the open source term I saw when trying to read the paper, I was not talking about EXACT reproducible builds, which I agree are probably a stretch with current tech (but obviously not impossible with new tech but possibly slower and not obviously needed anyway). I also know about OpenLLaMA and others but that's not Meta's work, and it's not the work in the paper. I'm pretty sure a lot of the core principles Llama uses are drawn from OpenAI's published research but I agree that there is a lack of openness recently, e.g people having to speculate about even core things like how they presumably use mixture of experts, but "open source" is not the right term for them just because OpenAI is not being open right now. I agree mostly that the model itself is somewhat open but I really think we should consider "open weights" as a much better term, I think it's much more reasonable than saying open source.

espadrine 3y ago

> Many clearly think of the weights as a source-like artifact. I believe releasing the weights is necessary for an open-source AI project, but don’t consider the weights to be source. Indeed, open source is not just about the code. For instance, the OSI definition[0] states: > The license must allow modifications and derived works If a company released source code for a project, but it was written in a new language whose compiler they held private, then making derived works would be extremely hard. Someone would have to reverse engineer how the language works from source examples, and reimplement a compiler with a similar performance. In that case, in my view, while the code was open, the project would not be open-source. The same is true of AI models: people can certainly reverse-engineer the training code and replicate a training run to get the weights, but without the weights, the project is not open-source. On the flip side, if the inference code is open and has the weights public but the training code is not, it is similar to an open-source project using a proprietary compiler (think C# before 2014). The project is open-source, but the training is not. [0]: https://opensource.org/osd/ https://opensource.org/osd/