2 ms·

I don’t think that counts as open source. They didn’t share any details about their training, making it basically impossible to replicate. It’s more akin to a

I don’t think that counts as open source. They didn’t share any details about their training, making it basically impossible to replicate. It’s more akin to a SaaS company releasing a compiled binary that usually runs on their server. Better than nothing, but not exactly in the spirit of open source. This doesn’t seem like a pedantic distinction, but I suppose it’s up to the community to agree or disagree.

minimaxir 3y ago
It's IMO a pedantic distinction. A compiled binary is a bad metaphor because it gives the implication that Mistral-7B is an as-is WYSIWIG project that's not easily modifiable. In contrast, there have been a bunch of new powerful new models created by modifying or finetuning Mistral-7B such as Zephyr-7B: https://huggingface.co/HuggingFaceH4/zephyr-7b-beta https://huggingface.co/HuggingFaceH4/zephyr-7b-beta The better analogy to Mistral-7B is something like modding Minecraft or Skyrim: although those games are closed source themselves, it has enabled innovations which helps the open-source community directly. It would be nice to have fully open-source methodologies but lacking them isn't an inherent disqualifier.
- hedgehog 3y ago
  It's a big distinction, if I want to tinker with the model architecture I essentially can't because the training pipeline is not public.
  - minimaxir 3y ago
    If you want to tinker with the architecture Hugging Face has a FOSS implementation in transformers: https://github.com/huggingface/transformers/blob/main/src/transformers/models/mistral/modeling_mistral.py https://github.com/huggingface/transformers/blob/main/src/tr... If you want to reproduce the training pipeline, you couldn't do that even if you wanted to because you don't have access to thousands of A100s.
    - hedgehog 3y ago
      I'm well aware of the many open source architectures, and the point stands. Models like GPT-J have open code and data, and that allows using them as a baseline for architecture experiments in a way that Mistral's models can't be. Mistral publishes weights and code, but not the training procedure or data. Not open.
    - sillysaurusx 3y ago
      We do, via TRC. Eleuther does too. I think it’s a bad idea to have a fatalistic attitude towards model reproduction.
      - hedgehog 3y ago
        Exactly, nice work BTW. And no hate for Mistral, they're doing great work, but let's not confuse weights-available with fully open models.
      - emadm 3y ago
        With all the new national supercomputers scale isn’t really going to be an issue, they all want large language models on 10k GH200s or whatever and the libraries are getting easier to use
mrob 3y ago
According to the Free Software Definition: "Source code is defined as the preferred form of the program for making changes in. Thus, whatever form a developer changes to develop the program is the source code of that developer's version." According to the Open Source Definition: "The source code must be the preferred form in which a programmer would modify the program. Deliberately obfuscated source code is not allowed. Intermediate forms such as the output of a preprocessor or translator are not allowed." LLM models are usually modified by changing the model weights directly, instead of retraining the model from scratch. LLM weights are poorly understood, but this is an unavoidable side effect of the development methodology, not deliberate obfuscation. "Intermediate" implies a form must undergo further processing before it can be used, but LLM weights are typically used directly. LLMs did not exist when these definitions were written, so they aren't a perfect fit for the terminology used, but there's a reasonable argument to be made that LLM weights can qualify as "source code".
- lmm 3y ago
  > LLM models are usually modified by changing the model weights directly, instead of retraining the model from scratch. LLM weights are poorly understood, but this is an unavoidable side effect of the development methodology, not deliberate obfuscation. They're understood based on knowing the training process though, and a developer working on them would want to have the option of doing a partial or full retraining where warranted.

minimaxir 3y ago

It's IMO a pedantic distinction. A compiled binary is a bad metaphor because it gives the implication that Mistral-7B is an as-is WYSIWIG project that's not easily modifiable. In contrast, there have been a bunch of new powerful new models created by modifying or finetuning Mistral-7B such as Zephyr-7B: https://huggingface.co/HuggingFaceH4/zephyr-7b-beta https://huggingface.co/HuggingFaceH4/zephyr-7b-beta The better analogy to Mistral-7B is something like modding Minecraft or Skyrim: although those games are closed source themselves, it has enabled innovations which helps the open-source community directly. It would be nice to have fully open-source methodologies but lacking them isn't an inherent disqualifier.

hedgehog 3y ago
It's a big distinction, if I want to tinker with the model architecture I essentially can't because the training pipeline is not public.
- minimaxir 3y ago
  If you want to tinker with the architecture Hugging Face has a FOSS implementation in transformers: https://github.com/huggingface/transformers/blob/main/src/transformers/models/mistral/modeling_mistral.py https://github.com/huggingface/transformers/blob/main/src/tr... If you want to reproduce the training pipeline, you couldn't do that even if you wanted to because you don't have access to thousands of A100s.
  - hedgehog 3y ago
    I'm well aware of the many open source architectures, and the point stands. Models like GPT-J have open code and data, and that allows using them as a baseline for architecture experiments in a way that Mistral's models can't be. Mistral publishes weights and code, but not the training procedure or data. Not open.
  - sillysaurusx 3y ago
    We do, via TRC. Eleuther does too. I think it’s a bad idea to have a fatalistic attitude towards model reproduction.
    - hedgehog 3y ago
      Exactly, nice work BTW. And no hate for Mistral, they're doing great work, but let's not confuse weights-available with fully open models.
    - emadm 3y ago
      With all the new national supercomputers scale isn’t really going to be an issue, they all want large language models on 10k GH200s or whatever and the libraries are getting easier to use

mrob 3y ago

According to the Free Software Definition: "Source code is defined as the preferred form of the program for making changes in. Thus, whatever form a developer changes to develop the program is the source code of that developer's version." According to the Open Source Definition: "The source code must be the preferred form in which a programmer would modify the program. Deliberately obfuscated source code is not allowed. Intermediate forms such as the output of a preprocessor or translator are not allowed." LLM models are usually modified by changing the model weights directly, instead of retraining the model from scratch. LLM weights are poorly understood, but this is an unavoidable side effect of the development methodology, not deliberate obfuscation. "Intermediate" implies a form must undergo further processing before it can be used, but LLM weights are typically used directly. LLMs did not exist when these definitions were written, so they aren't a perfect fit for the terminology used, but there's a reasonable argument to be made that LLM weights can qualify as "source code".

lmm 3y ago
> LLM models are usually modified by changing the model weights directly, instead of retraining the model from scratch. LLM weights are poorly understood, but this is an unavoidable side effect of the development methodology, not deliberate obfuscation. They're understood based on knowing the training process though, and a developer working on them would want to have the option of doing a partial or full retraining where warranted.