6 ms·

Llama’s code is open source: https://github.com/meta-llama/llama3/blob/main/llama/model.py https://github.com/meta-llama/llama3/blob/main/llama/model.p...

apsec112 2y ago
That's not the training code, just the inference code. The training code, running on thousands of high-end H100 servers, is surely much more complex. They also don't open-source the dataset, or the code they used for data scraping/filtering/etc.
- the8thbit 2y ago
  "just the inference code" It's not the "inference code", its the code that specifies the architecture of the model and loads the model. The "inference code" is mostly the model, and the model is not legible to a human reader. Maybe someday open source models will be possible, but we will need much better interpretability tools so we can generate the source code from the model. In most software projects you write the source as a specification that is then used by the computer to implement the software, but in this case the process is reversed.
blackeyeblitzar 2y ago
That is just the inference code. Not training code or evaluation code or whatever pre/post processing they do.
- patrickaljord 2y ago
  Is there an LLM with actual open source training code and dataset? Besides BLOOM https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom
  - osanseviero 2y ago
    Yes, there are a few dozen full open source models (license, code, data, models)
    - blackeyeblitzar 2y ago
      What are some of the other ones? I am aware mainly of OLMo (https://blog.allenai.org/olmo-open-language-model-87ccfc95f580 https://blog.allenai.org/olmo-open-language-model-87ccfc95f5...)
  - navinsylvester 2y ago
    Here you go - https://github.com/apple/corenet https://github.com/apple/corenet
mesebrec 2y ago
This is like saying any python program is open source because the python runtime is open source. Inference code is the runtime; the code that runs the model. Not the model itself.
- mkolodny 2y ago
  I disagree. The file I linked to, model.py, contains the Llama 3 model itself. You can use that model with open data to train it from scratch yourself. Or you can load Meta’s open weights and have a working LLM.
  - causal 2y ago
    Yeah a lot of people here seem to not understand that PyTorch really does make model definitions that simple, and that has everything you need to resume back-propagation. Not to mention PyTorch itself being open-sourced by Meta. That said the LLama-license doesn't meet strict definitions of OS, and I bet they have internal tooling for datacenter-scale training that's not represented here.
  - yjftsjthsd-h 2y ago
    > The file I linked to, model.py, contains the Llama 3 model itself. That makes it source available ( https://en.wikipedia.org/wiki/Source-available_software https://en.wikipedia.org/wiki/Source-available_software ), not open source
    - macrolime 2y ago
      Source available means you can see the source, but not modify it. This is kinda the opposite, you can modify the model, but you don't see all the details of its creation.
      - yjftsjthsd-h 2y ago
        > Source available means you can see the source, but not modify it. No, it doesn't mean that. To quote the page I linked, emphasis mine, > Source-available software is software released through a source code distribution model that includes arrangements where the source can be viewed, and in some cases modified, but without necessarily meeting the criteria to be called open-source. The licenses associated with the offerings range from allowing code to be viewed for reference to allowing code to be modified and redistributed for both commercial and non-commercial purposes. > This is kinda the opposite, you can modify the model, but you don't see all the details of its creation. Per https://github.com/meta-llama/llama3/blob/main/LICENSE https://github.com/meta-llama/llama3/blob/main/LICENSE there's also a laundry list of ways you're not allowed to use it, including restrictions on commercial use. So not Open Source.
Flimm 2y ago
No, it's not. The Llama 3 Community License Agreement is not an open source license. Open source licenses need to meet the criteria of the only widely accepted definition of "open source", and that's the one formulated by the OSI [0]. This license has multiple restrictions on use and distribution which make it not open source. I know Facebook keeps calling this stuff open source, maybe in order to get all the good will that open source branding gets you, but that doesn't make it true. It's like a company calling their candy vegan while listing one its ingredients as pork-based gelatin. No matter how many times the company advertises that their product is vegan, it's not, because it doesn't meet the definition of vegan. [0] - https://opensource.org/osd https://opensource.org/osd
- CamperBob2 2y ago
  Open source licenses need to meet the criteria of the only widely accepted definition of "open source", and that's the one formulated by the OSI [0] Who died and made OSI God?
  - vbarrielle 2y ago
    The OSI was created about 20 years ago and defined and popularized the term open source. Their definition has been widely accepted over that period. Recently, companies are trying to market things as open source when in reality, they fail to adhere to the definition. I think we should not let these companies change the meaning of the term, which means it's important to explain every time they try to seem more open than they are. I'm afraid the battle is being lost though.
    - Suppafly 2y ago
      >The OSI was created about 20 years ago and defined and popularized the term open source. Their definition has been widely accepted over that period. It was defined and accepted by the community well before OSI came around though.
      - gowld 2y ago

apsec112 2y ago

That's not the training code, just the inference code. The training code, running on thousands of high-end H100 servers, is surely much more complex. They also don't open-source the dataset, or the code they used for data scraping/filtering/etc.

the8thbit 2y ago
"just the inference code" It's not the "inference code", its the code that specifies the architecture of the model and loads the model. The "inference code" is mostly the model, and the model is not legible to a human reader. Maybe someday open source models will be possible, but we will need much better interpretability tools so we can generate the source code from the model. In most software projects you write the source as a specification that is then used by the computer to implement the software, but in this case the process is reversed.

blackeyeblitzar 2y ago

That is just the inference code. Not training code or evaluation code or whatever pre/post processing they do.

patrickaljord 2y ago
Is there an LLM with actual open source training code and dataset? Besides BLOOM https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom
- osanseviero 2y ago
  Yes, there are a few dozen full open source models (license, code, data, models)
  - blackeyeblitzar 2y ago
    What are some of the other ones? I am aware mainly of OLMo (https://blog.allenai.org/olmo-open-language-model-87ccfc95f580 https://blog.allenai.org/olmo-open-language-model-87ccfc95f5...)
- navinsylvester 2y ago
  Here you go - https://github.com/apple/corenet https://github.com/apple/corenet

mesebrec 2y ago

This is like saying any python program is open source because the python runtime is open source. Inference code is the runtime; the code that runs the model. Not the model itself.

mkolodny 2y ago
I disagree. The file I linked to, model.py, contains the Llama 3 model itself. You can use that model with open data to train it from scratch yourself. Or you can load Meta’s open weights and have a working LLM.
- causal 2y ago
  Yeah a lot of people here seem to not understand that PyTorch really does make model definitions that simple, and that has everything you need to resume back-propagation. Not to mention PyTorch itself being open-sourced by Meta. That said the LLama-license doesn't meet strict definitions of OS, and I bet they have internal tooling for datacenter-scale training that's not represented here.
- yjftsjthsd-h 2y ago
  > The file I linked to, model.py, contains the Llama 3 model itself. That makes it source available ( https://en.wikipedia.org/wiki/Source-available_software https://en.wikipedia.org/wiki/Source-available_software ), not open source
  - macrolime 2y ago
    Source available means you can see the source, but not modify it. This is kinda the opposite, you can modify the model, but you don't see all the details of its creation.
    - yjftsjthsd-h 2y ago
      > Source available means you can see the source, but not modify it. No, it doesn't mean that. To quote the page I linked, emphasis mine, > Source-available software is software released through a source code distribution model that includes arrangements where the source can be viewed, and in some cases modified, but without necessarily meeting the criteria to be called open-source. The licenses associated with the offerings range from allowing code to be viewed for reference to allowing code to be modified and redistributed for both commercial and non-commercial purposes. > This is kinda the opposite, you can modify the model, but you don't see all the details of its creation. Per https://github.com/meta-llama/llama3/blob/main/LICENSE https://github.com/meta-llama/llama3/blob/main/LICENSE there's also a laundry list of ways you're not allowed to use it, including restrictions on commercial use. So not Open Source.

Flimm 2y ago

No, it's not. The Llama 3 Community License Agreement is not an open source license. Open source licenses need to meet the criteria of the only widely accepted definition of "open source", and that's the one formulated by the OSI [0]. This license has multiple restrictions on use and distribution which make it not open source. I know Facebook keeps calling this stuff open source, maybe in order to get all the good will that open source branding gets you, but that doesn't make it true. It's like a company calling their candy vegan while listing one its ingredients as pork-based gelatin. No matter how many times the company advertises that their product is vegan, it's not, because it doesn't meet the definition of vegan. [0] - https://opensource.org/osd https://opensource.org/osd

CamperBob2 2y ago
Open source licenses need to meet the criteria of the only widely accepted definition of "open source", and that's the one formulated by the OSI [0] Who died and made OSI God?
- vbarrielle 2y ago
  The OSI was created about 20 years ago and defined and popularized the term open source. Their definition has been widely accepted over that period. Recently, companies are trying to market things as open source when in reality, they fail to adhere to the definition. I think we should not let these companies change the meaning of the term, which means it's important to explain every time they try to seem more open than they are. I'm afraid the battle is being lost though.
  - Suppafly 2y ago
    >The OSI was created about 20 years ago and defined and popularized the term open source. Their definition has been widely accepted over that period. It was defined and accepted by the community well before OSI came around though.
    - gowld 2y ago