7 ms·

Why did Meta open-source Llama 2?

73 points by rckrd 3y ago

seba_dos1 3y ago
They didn't. It's not open source.
- throwaway29812 3y ago
  can you expand on that?
  - onedognight 3y ago
    It’s open but not open source. Getting Llama is like getting a binary executable. You don’t get the source (training data / model / code), so you can’t make changes and recompile, but you can use it and fine tune.
    - jraph 3y ago
      What does open mean? I'm not sure we really have a widely shared, common definition of this, contrary to open source. > You don’t get the source You do IIUC. But your can't use it for all purposes (like serving 1B users), which is what makes it non open source.
  - seba_dos1 3y ago
    Its license is quite obviously not matching the definition of Open Source: > v. You will not use the Llama Materials or any output or results of the Llama Materials to improve any other large language model (excluding Llama 2 or derivative works thereof). > 2. Additional Commercial Terms. If, on the Llama 2 version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Meta, which Meta may grant to you in its sole discretion, and you are not authorized to exercise any of the rights under this Agreement unless or until Meta otherwise expressly grants you such rights. > Prohibited Uses: <a whole page full of text>
  - andy99 3y ago
    The license has a bunch of terms and restrictions that make it incompatible with accepted definitions of open source software. LLaMA2 isn't "Open Source" - and why it doesn't matter https://www.alessiofanelli.com/blog/llama2-isnt-open-source https://www.alessiofanelli.com/blog/llama2-isnt-open-source Meta can call Llama open source as much as it likes, but that doesn't mean it is https://www.theregister.com/2023/07/21/llama_is_not_open_source/ https://www.theregister.com/2023/07/21/llama_is_not_open_sou... Software licenses masquerading as open source (I wrote that one) http://marble.onl/posts/software-licenses-masquerading-as-open-source.html http://marble.onl/posts/software-licenses-masquerading-as-op... There have been lots of other posts on here about this too.
  - mindcrime 3y ago
    It's closer to "shared source" or "source available". The license is not compliant with the OSD[1] which is the de-facto (but not de jure, for you pedants out there) definition of "Open Source" in common usage. [1]: https://opensource.org/osd/ https://opensource.org/osd/
    - jraph 3y ago
      > but not de jure, for you pedants out there This means that the term "open source" is not legally defined (and not recognized by lawyer / judges) [and the OSD], right?
      - mindcrime 3y ago
        Yes, "de jure" more or less translates to "by law", and "de facto" is something like "by practice" or "by convention". So the Open Source Initiative folks have no legal basis to enforce use of their definition, but in practice it's so widely used and acknowledged that for all practical purposes it is the definition.
  - _Parfait_ 3y ago
    Basically under the actual terms of what open source is there can be no limits to particular users. Meta, in order to protect their competitive market share has said explicitly in Paragraph Two of their licensing terms --- " If, on the Llama 2 version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Meta, which Meta may grant to you in its sole discretion. -- Doing so, technically, takes the license out of the category of “Open Source.” Sources: Open Source Initiate Argument https://blog.opensource.org/metas-llama-2-license-is-not-open-source/ https://blog.opensource.org/metas-llama-2-license-is-not-ope... Llama Terms https://ai.meta.com/resources/models-and-libraries/llama-downloads/ https://ai.meta.com/resources/models-and-libraries/llama-dow...
    - wolpoli 3y ago
      I wonder who they are targeting with this restriction on 700 million MAU.
      - maattp 3y ago
        Snapchat
      - _Parfait_ 3y ago
        The big boys, MS & Big G
- stale2002 3y ago
  We don't have to play dumb. Everyone in this thread knows exactly what the article means, when it says that Llama was open sourced. This obviously means that the weights were released, and everyone knows this, regardless of any pedantic definition of what "open source" means.
  - seba_dos1 3y ago
    Calling Llama 2 "open source" is obviously a misleading marketing tactic to capitalize on positive connotations of the term, while in reality it's nowhere even close to being open source. This has nothing to do with any kind of pedantry, "open source" is a specific term with specific meaning, and "releasing the weights" is completely orthogonal to it being open source or not.
    - andy99 3y ago
      A challenge here is that it is pedantic, even if it's true. I've been very vocal about how the LLaMAs as well as other models like those released under RAIL licenses are not open source. But the truth is very few people care and attitudes like the one above are common. What needs to happen is for a strong, and I'd suggest copyleft, alternative to emerge that's good enough to make people have to care more about license terms. That plus continued advocacy in the face of people dismissing pedantic distinctions as irrelevant.
      - nomel 3y ago
        > and I'd suggest copyleft, alternative to emerge that's good enough to make people have to care more about license terms I keep seeing this, but I can't understand how it would work. Bored nerds will volunteer their time to open source, with little return. That's why I did it. Generally, nobody will volunteer their money to open source. This is unfortunate, and a huge problem. Training neural nets requires the money. Where does it come from?
        andy99 3y ago
        I'm not saying it's a solved problem though there are already initiatives like AI Horde that are doing volunteer distributed inference (it it looks like now fine-tuning) on GPUs https://stablehorde.net/ https://stablehorde.net/ Plus it can be a strategic choice for institutions and companies, just like Linux is for example.
- topynate 3y ago
  They can keep pretending it's an open source licence, and I'll keep pretending to respect their copyright in algorithmically generated weights.
- mrtranscendence 3y ago
  Definitions are determined by actual usage. If people call Llama 2’s license open source then ipso facto it’s open source.
  - seba_dos1 3y ago
    There's about 25 years of actual usage that clearly states otherwise. We can give your idea some weight after 25 years of people consistently calling Llama 2 open source, effectively diluting the term to mean nothing in the process. Meanwhile it's just as if I told you that the sky is green.
    - mrtranscendence 3y ago
      There’s a history of actual usage among some subsets of people, not for everyone. Regardless, it doesn’t really matter if there’s history as long as usage is in fact changing. “Literally” meant one thing much longer than it has been a superlative.
      - seba_dos1 3y ago
        I said above that the sky is green, just like people from Meta said that Llama 2 is open source. See! Actual usage! The sky is now green. The only reason people say that Llama 2 is open source is because they have been misled, not because the definition is changing. There's a lot of goodwill around the term "open source" because of its actual meaning, so it will always be tempting to abuse it for your own benefit, as not everyone bothers to check whether your claims are actually true.
andy99 3y ago
Llama had already become a defacto standard for LLMs, between all the fine-tunes and llama.cpp. Giving it a wider license really cements it as a standard while making everybody use it on Meta's terms. It's an "open source" strategy where all the benefits accrue back to Meta instead of the community but that's permissive enough to placate most people. Personally I feel like it's a dangerous precedent because it shifts open source from a community concept to something that a company lets you have with a bunch of conditions.
- brucethemoose2 3y ago
  Stable Diffusion is a good example of this progression. There are new models/papers/backends coming out all the time (including SDXL, which is similar to the original SD), but the community is so entrenched in Stable Diffusion 1.5 and the old Stability AI PyTorch implementation that moving to anything else hasn't really happened yet. That being said, I think Llama won't stick around as the "de facto" standard for over a year unless Meta keeps releasing better foundational models.
  - pavlov 3y ago
    Why wouldn’t they? Meta seems to have the money, the hardware, the talent, and the executive motivation to keep producing better models.
  - nullc 3y ago
    > but the community is so entrenched in Stable Diffusion 1.5 and the old Stability AI PyTorch implementation that moving to anything else hasn't really happened yet. It's not just inertia, the later models (even 1.5 over 1.4) are heavily censored. And at least in the case of 1.5 it appears to cause obviously inferior output even for content which is in no way explicit. My understanding is the facebook published Llama 2 chat fine tunes are more or less the most censored LLM yet, refusing to even discuss negative sentiments. I bet we'll see other people's RLHF fine tunes on the base model being used as a point for more developent. It's an unfortunate development that it appears that the primary commercial "value add" is building models that don't do what the users unambiguously direct them to do. It's hard to make a model smarter, but easier to lobotomize one.
phkahler 3y ago
To increase adoption. They are also working with Qualcom [1] to bring it on-device. Not sure if they're licensing it, but when you tweak hardware for something specific, you kinda want people to use it. [1] https://www.qualcomm.com/news/releases/2023/07/qualcomm-works-with-meta-to-enable-on-device-ai-applications-usi https://www.qualcomm.com/news/releases/2023/07/qualcomm-work...
innagadadavida 3y ago
All the responses I'm reading so far are rather shallow and fails to consider the overall landscape and how it will evolve and who the big losers will be. The way I see it, that the current players include Google (which has a lot to lose), OpenAI (unclear business model) and upcoming startups (can disrupt Google/OpenAI). Meta releasing these models will impact Google and OpenAI the most by helping upcoming startups to inflict fatal blows or slowly chip away at their business models by means of a race to the bottom. The main issue preventing Google or OpenAI to succeed is that the regulatory landscape will pose a huge risk and Meta knows that. Startups are not hampered by this as they are small fry and before anyone can notice, they can/will land a blow on Google/OpenAI. To all those people complaining on this not being open source - Zuck is playing chess, while you play a much simpler game. Advancing SOTA and a bit of Open source is a side benefit.
- seba_dos1 3y ago
  I don't care whether it's open source or not. I do care about it being called open source when it's not.
  - jraph 3y ago
    > I do care about it being called open source when it's not. Yep. We don't need additional confusion on this term.
m3kw9 3y ago
This analogous to freeium model for apps?
lincon127 3y ago
Did it? I don't think they did
cloudking 3y ago
They are taking a similar AI strategy to Google's mobile strategy with Android.
deleted 3y ago
[deleted]
Tepix 3y ago
So, rckrd wrote a (rather short) article about the license of Llama 2 but got it all wrong: Even though the press calls it open-source, it's not. Open Source has a very clear definition. Llama 2 fails in multiple regards. First, the license by itself is not an open source license. It has important restrictions that make it non-open source. Second, the distribution. You have to apply for the download with a web form and you are not allowed to redistribute the model. Third, the source code, i.e. the data used to train the model. Meta is not telling us about it. One of the core features of open source is that you can recreate the binary (in this case, the model weights) by yourself. You can't do that here.
- charcircuit 3y ago
  You can redistribute the model. A model is not a binary. Training code is not source code for the model. The training code is closer to an editor. You can release open source software created by a closed source IDE.
  - Tepix 3y ago
    > You can redistribute the model. That's great. I disagree about the other part.
  - catchnear4321 3y ago
    the source of the model is code and data. you cannot generate the model without the same code and training data. it is not open source. you are conflating editing the model with the capability to edit a thing. using your source code example, altering source code changes the binary. as if one were editing the binary. the source code isn’t called an IDE.
  - fragmede 3y ago
    What's the definition of binary you're using here? Because a model is an inscrutable blob of bytes that is used (with the help of libaries and other glue code) to perform a function.
- taneq 3y ago
  Are we gonna have to do the FOSS vs. 'source available' thing again? Also, while I think your 'source=training set+model, binary=weights' analysis is correct, I'm not sure we have 100% consensus on this yet?
dwrodri 3y ago
Meta made Llama 2 source-and-weights available because they agreed with the observations in the leaked Google memo[1]. Meta got a huge amount of infra/research/experimentation work done on top of LLaMa. Pre-training wasn’t cheap, but they got datapoints no one else in big tech had, abd that is very valuable, especially when building a bridge into a new frontier like LLM-driven products.
frankreyes 3y ago
Divide et impera

73 points by rckrd 3y ago

seba_dos1 3y ago

They didn't. It's not open source.

throwaway29812 3y ago
can you expand on that?
- onedognight 3y ago
  It’s open but not open source. Getting Llama is like getting a binary executable. You don’t get the source (training data / model / code), so you can’t make changes and recompile, but you can use it and fine tune.
  - jraph 3y ago
    What does open mean? I'm not sure we really have a widely shared, common definition of this, contrary to open source. > You don’t get the source You do IIUC. But your can't use it for all purposes (like serving 1B users), which is what makes it non open source.
- seba_dos1 3y ago
  Its license is quite obviously not matching the definition of Open Source: > v. You will not use the Llama Materials or any output or results of the Llama Materials to improve any other large language model (excluding Llama 2 or derivative works thereof). > 2. Additional Commercial Terms. If, on the Llama 2 version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Meta, which Meta may grant to you in its sole discretion, and you are not authorized to exercise any of the rights under this Agreement unless or until Meta otherwise expressly grants you such rights. > Prohibited Uses: <a whole page full of text>
- andy99 3y ago
  The license has a bunch of terms and restrictions that make it incompatible with accepted definitions of open source software. LLaMA2 isn't "Open Source" - and why it doesn't matter https://www.alessiofanelli.com/blog/llama2-isnt-open-source https://www.alessiofanelli.com/blog/llama2-isnt-open-source Meta can call Llama open source as much as it likes, but that doesn't mean it is https://www.theregister.com/2023/07/21/llama_is_not_open_source/ https://www.theregister.com/2023/07/21/llama_is_not_open_sou... Software licenses masquerading as open source (I wrote that one) http://marble.onl/posts/software-licenses-masquerading-as-open-source.html http://marble.onl/posts/software-licenses-masquerading-as-op... There have been lots of other posts on here about this too.
- mindcrime 3y ago
  It's closer to "shared source" or "source available". The license is not compliant with the OSD[1] which is the de-facto (but not de jure, for you pedants out there) definition of "Open Source" in common usage. [1]: https://opensource.org/osd/ https://opensource.org/osd/
  - jraph 3y ago
    > but not de jure, for you pedants out there This means that the term "open source" is not legally defined (and not recognized by lawyer / judges) [and the OSD], right?
    - mindcrime 3y ago
      Yes, "de jure" more or less translates to "by law", and "de facto" is something like "by practice" or "by convention". So the Open Source Initiative folks have no legal basis to enforce use of their definition, but in practice it's so widely used and acknowledged that for all practical purposes it is the definition.
- _Parfait_ 3y ago
  Basically under the actual terms of what open source is there can be no limits to particular users. Meta, in order to protect their competitive market share has said explicitly in Paragraph Two of their licensing terms --- " If, on the Llama 2 version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Meta, which Meta may grant to you in its sole discretion. -- Doing so, technically, takes the license out of the category of “Open Source.” Sources: Open Source Initiate Argument https://blog.opensource.org/metas-llama-2-license-is-not-open-source/ https://blog.opensource.org/metas-llama-2-license-is-not-ope... Llama Terms https://ai.meta.com/resources/models-and-libraries/llama-downloads/ https://ai.meta.com/resources/models-and-libraries/llama-dow...
  - wolpoli 3y ago
    I wonder who they are targeting with this restriction on 700 million MAU.
    - maattp 3y ago
      Snapchat
    - _Parfait_ 3y ago
      The big boys, MS & Big G
stale2002 3y ago
We don't have to play dumb. Everyone in this thread knows exactly what the article means, when it says that Llama was open sourced. This obviously means that the weights were released, and everyone knows this, regardless of any pedantic definition of what "open source" means.
- seba_dos1 3y ago
  Calling Llama 2 "open source" is obviously a misleading marketing tactic to capitalize on positive connotations of the term, while in reality it's nowhere even close to being open source. This has nothing to do with any kind of pedantry, "open source" is a specific term with specific meaning, and "releasing the weights" is completely orthogonal to it being open source or not.
  - andy99 3y ago
    A challenge here is that it is pedantic, even if it's true. I've been very vocal about how the LLaMAs as well as other models like those released under RAIL licenses are not open source. But the truth is very few people care and attitudes like the one above are common. What needs to happen is for a strong, and I'd suggest copyleft, alternative to emerge that's good enough to make people have to care more about license terms. That plus continued advocacy in the face of people dismissing pedantic distinctions as irrelevant.
    - nomel 3y ago
      > and I'd suggest copyleft, alternative to emerge that's good enough to make people have to care more about license terms I keep seeing this, but I can't understand how it would work. Bored nerds will volunteer their time to open source, with little return. That's why I did it. Generally, nobody will volunteer their money to open source. This is unfortunate, and a huge problem. Training neural nets requires the money. Where does it come from?
      - andy99 3y ago
        I'm not saying it's a solved problem though there are already initiatives like AI Horde that are doing volunteer distributed inference (it it looks like now fine-tuning) on GPUs https://stablehorde.net/ https://stablehorde.net/ Plus it can be a strategic choice for institutions and companies, just like Linux is for example.
topynate 3y ago
They can keep pretending it's an open source licence, and I'll keep pretending to respect their copyright in algorithmically generated weights.
mrtranscendence 3y ago
Definitions are determined by actual usage. If people call Llama 2’s license open source then ipso facto it’s open source.
- seba_dos1 3y ago
  There's about 25 years of actual usage that clearly states otherwise. We can give your idea some weight after 25 years of people consistently calling Llama 2 open source, effectively diluting the term to mean nothing in the process. Meanwhile it's just as if I told you that the sky is green.
  - mrtranscendence 3y ago
    There’s a history of actual usage among some subsets of people, not for everyone. Regardless, it doesn’t really matter if there’s history as long as usage is in fact changing. “Literally” meant one thing much longer than it has been a superlative.
    - seba_dos1 3y ago
      I said above that the sky is green, just like people from Meta said that Llama 2 is open source. See! Actual usage! The sky is now green. The only reason people say that Llama 2 is open source is because they have been misled, not because the definition is changing. There's a lot of goodwill around the term "open source" because of its actual meaning, so it will always be tempting to abuse it for your own benefit, as not everyone bothers to check whether your claims are actually true.

andy99 3y ago

Llama had already become a defacto standard for LLMs, between all the fine-tunes and llama.cpp. Giving it a wider license really cements it as a standard while making everybody use it on Meta's terms. It's an "open source" strategy where all the benefits accrue back to Meta instead of the community but that's permissive enough to placate most people. Personally I feel like it's a dangerous precedent because it shifts open source from a community concept to something that a company lets you have with a bunch of conditions.

brucethemoose2 3y ago
Stable Diffusion is a good example of this progression. There are new models/papers/backends coming out all the time (including SDXL, which is similar to the original SD), but the community is so entrenched in Stable Diffusion 1.5 and the old Stability AI PyTorch implementation that moving to anything else hasn't really happened yet. That being said, I think Llama won't stick around as the "de facto" standard for over a year unless Meta keeps releasing better foundational models.
- pavlov 3y ago
  Why wouldn’t they? Meta seems to have the money, the hardware, the talent, and the executive motivation to keep producing better models.
- nullc 3y ago
  > but the community is so entrenched in Stable Diffusion 1.5 and the old Stability AI PyTorch implementation that moving to anything else hasn't really happened yet. It's not just inertia, the later models (even 1.5 over 1.4) are heavily censored. And at least in the case of 1.5 it appears to cause obviously inferior output even for content which is in no way explicit. My understanding is the facebook published Llama 2 chat fine tunes are more or less the most censored LLM yet, refusing to even discuss negative sentiments. I bet we'll see other people's RLHF fine tunes on the base model being used as a point for more developent. It's an unfortunate development that it appears that the primary commercial "value add" is building models that don't do what the users unambiguously direct them to do. It's hard to make a model smarter, but easier to lobotomize one.

phkahler 3y ago

To increase adoption. They are also working with Qualcom [1] to bring it on-device. Not sure if they're licensing it, but when you tweak hardware for something specific, you kinda want people to use it. [1] https://www.qualcomm.com/news/releases/2023/07/qualcomm-works-with-meta-to-enable-on-device-ai-applications-usi https://www.qualcomm.com/news/releases/2023/07/qualcomm-work...

innagadadavida 3y ago

All the responses I'm reading so far are rather shallow and fails to consider the overall landscape and how it will evolve and who the big losers will be. The way I see it, that the current players include Google (which has a lot to lose), OpenAI (unclear business model) and upcoming startups (can disrupt Google/OpenAI). Meta releasing these models will impact Google and OpenAI the most by helping upcoming startups to inflict fatal blows or slowly chip away at their business models by means of a race to the bottom. The main issue preventing Google or OpenAI to succeed is that the regulatory landscape will pose a huge risk and Meta knows that. Startups are not hampered by this as they are small fry and before anyone can notice, they can/will land a blow on Google/OpenAI. To all those people complaining on this not being open source - Zuck is playing chess, while you play a much simpler game. Advancing SOTA and a bit of Open source is a side benefit.

seba_dos1 3y ago
I don't care whether it's open source or not. I do care about it being called open source when it's not.
- jraph 3y ago
  > I do care about it being called open source when it's not. Yep. We don't need additional confusion on this term.

m3kw9 3y ago

This analogous to freeium model for apps?

lincon127 3y ago

Did it? I don't think they did

cloudking 3y ago

They are taking a similar AI strategy to Google's mobile strategy with Android.

deleted 3y ago

[deleted]

Tepix 3y ago

So, rckrd wrote a (rather short) article about the license of Llama 2 but got it all wrong: Even though the press calls it open-source, it's not. Open Source has a very clear definition. Llama 2 fails in multiple regards. First, the license by itself is not an open source license. It has important restrictions that make it non-open source. Second, the distribution. You have to apply for the download with a web form and you are not allowed to redistribute the model. Third, the source code, i.e. the data used to train the model. Meta is not telling us about it. One of the core features of open source is that you can recreate the binary (in this case, the model weights) by yourself. You can't do that here.

charcircuit 3y ago
You can redistribute the model. A model is not a binary. Training code is not source code for the model. The training code is closer to an editor. You can release open source software created by a closed source IDE.
- Tepix 3y ago
  > You can redistribute the model. That's great. I disagree about the other part.
- catchnear4321 3y ago
  the source of the model is code and data. you cannot generate the model without the same code and training data. it is not open source. you are conflating editing the model with the capability to edit a thing. using your source code example, altering source code changes the binary. as if one were editing the binary. the source code isn’t called an IDE.
- fragmede 3y ago
  What's the definition of binary you're using here? Because a model is an inscrutable blob of bytes that is used (with the help of libaries and other glue code) to perform a function.
taneq 3y ago
Are we gonna have to do the FOSS vs. 'source available' thing again? Also, while I think your 'source=training set+model, binary=weights' analysis is correct, I'm not sure we have 100% consensus on this yet?

dwrodri 3y ago

Meta made Llama 2 source-and-weights available because they agreed with the observations in the leaked Google memo[1]. Meta got a huge amount of infra/research/experimentation work done on top of LLaMa. Pre-training wasn’t cheap, but they got datapoints no one else in big tech had, abd that is very valuable, especially when building a bridge into a new frontier like LLM-driven products.

frankreyes 3y ago

Divide et impera