4 ms·

For a model to be ‘fully’ open source you need more than the model itself and a way to run it. You also need the data and the program that can be used to train

For a model to be ‘fully’ open source you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it. See The Open Source AI Definition from OSI: https://opensource.org/ai https://opensource.org/ai

szundi 2y ago
Yes. I cannot comprehend this to this day. A model weights data + runner is how different from a closed source executable? Why do everyone call these open source?
- fhd2 2y ago
  Because practically speaking, you can fine tune them I suppose? But that's also true for binaries, games are a good example of where people pushed this quite far. Based on what little experience I have in ML, I'd say it's about the same thing. Whereas an API is more akin to a piece of software you can't tinker with in any way. Guess the bar is just lower in the LLM space :P
  - darkwater 2y ago
    So a proprietary program with a lot of knobs and configuration files is kind of opensource?
    - fhd2 2y ago
      By what appears to be the logic for "open source AI", a locally executable proprietary program would be "open source" (because you can meddle with the executable). To me, that's mostly just "not SaaS". But somehow, a different definition appears to have stuck for LLMs than for other types of software.
- PeterStuer 2y ago
  Because typically adapting or improving traditional code to your needs is very difficult without access to the source code and build files. For an LLM you can finetune and enhance, distill and embed given just the model weights, the runtime, and a permissive license. Having more is better. Well written detailed model release papers help a lot. Training code and training data are a great bonus. However, I find the purity contest a bit too dismissive of the great contributions to the AI dev ecosystem that Meta and Deepseek have brought us. Without these, there wouldn't be the open ecosystem we have today.
  - GeneralMayhem 2y ago
    It's not a purity contest, it's a clarity contest. If Meta and Deepseek want to operate the way they have been, where they release baked models and whitepapers, that's fine - and you're right, it's certainly more than they're obligated to release. They just shouldn't be calling it "open source" when the source is literally not open.
    - doix 2y ago
      Eh, I can kinda see it. It depends on your definitions of words. People have been muddying the waters with what "open source" means anyway. I have known it to mean code released under an open source license. Other people use it to mean programs where the source is available regardless of license. I would use "source available" to describe that, but some people strongly disagree with my definitions. If I write a program, then obfuscate it and then release the obfuscated code under an open source license, would you consider it open source(I would)? That's kind of the case here, they are releasing the model weights under an open source license. Personally, I think it's fine to shorten it to "open source model" instead of "a model with the weights released under an open source license". What I would object to is releasing model weights under a restrictive license and calling that open source.
      - GeneralMayhem 2y ago
        > If I write a program, then obfuscate it and then release the obfuscated code under an open source license, would you consider it open source(I would)? I wouldn't. Most definitions of open source say something like "in the form used for editing". You can release a built binary under an unrestrictive license, but that does not mean that you've opened the source. It's literally the plain meaning of the words: the source, as in where the thing comes from, needs to be open for it be meaningful.
- sigmoid10 2y ago
  Open source becomes really complicated once assets with unclear license are involved in any way. Lots of people for example would say that Jedi Knight 2 is open source because Raven Software released the source code and tools needed to build the game. But that alone doesn't mean you can run it, because you still need to get a hold of all the assets (models, textures, sounds) which may or may not still be property of LucasArts or its successors. Even if you have them, it's actually unclear if it is legal to use them this way. So while there are tons of people working on mods and conversions, noone in their right mind would distribute all the source assets. Much in the same way, no sane company will touch the legal nightmare of releasing LLM training data scraped from public websites. Even releasing the LLM alone might be infringement, there are literally court cases being fought over this right now.
  - GeneralMayhem 2y ago
    Games like that, or the open-source clones of commercial games that require original assets to play (e.g. OpenXCOM), actually give a very clear analogy here: open source does not mean open assets. The software code is under a separate license from the data it processes. Emulators like Dolphin are kind of in this situation too - the program is open, the data it processes is not. And that's fine! It's still valuable to have access to the source code, even if the "batteries" aren't included. Of course, if you really want to call it an open source model you should include the source for the data scraping/cleaning stages too; then the only thing missing would be the compute time and risk of acquiring dubiously-legal inputs. I personally prefer a taxonomy like: * Open weights: you can download the artifact and run it locally, not just use it through an application like chatgpt or an API. * Open source: the code that created the artifact is provided in the same format that the authors used to work on it. * Open data: the dataset that the source code was used on is available for download. All three of those could be individually licensed or released, for 8 possible combinations. In the analogy to games, they would correspond to the licenses on the retail binary, the source code of the game, and the original uncompressed art assets or Blender projects, respectively.
    - sigmoid10 2y ago
      If it has already been established that open source doesn't mean open assets, why would we change that now? After all, training data is literally nothing but assets - except that you don't need them to run the application. So in that sense open LLMs are more open than these games.
- benterix 2y ago
  Because Meta called them in this way to differentiate themselves from "Open"AI and everyone else followed the suit.
- deleted 2y ago
  [deleted]
- moffkalast 2y ago
  Because marketing (open source is a buzzword after all), and the media just repeats what they read in press releases verbatim. But most people working with the models themselves call them open-weight, except for some occasional exception like OLMo that publishes the dataset and training scripts and is actually open source.
- dartos 2y ago
  Because Meta called llama that shortly after it got leaked and it stuck. The AI crowd doesn’t care much for licenses anyway.
rettichschnidi 2y ago
Sadly, the OSAID also does not require training data to be available. :(
a2128 2y ago
Is it reasonable to expect companies to redistribute 100TB of copyrighted content they used for their LLM, just on the off-chance someone has a few million laying around and wants to reproduce the model from scratch?
- anotherhue 2y ago
  The RL-only (no SFT) approaches might remove that issue. Problem sets should be smaller (and mechanically creatable) than the entire western corpus.
- puapuapuq 2y ago
  For open source model claiming SOTA performance, we could at least check for data leak from its training data.
- emseetech 2y ago
  Would a reference file with filename, size, source and checksum count towards the OSI definition?
- atq2119 2y ago
  Perhaps that's not reasonable to expect, but Meta apparently kind of did it anyway, if not in a way that helps reproduce their LLM: https://arstechnica.com/tech-policy/2025/02/meta-torrented-over-81-7tb-of-pirated-books-to-train-ai-authors-say/ https://arstechnica.com/tech-policy/2025/02/meta-torrented-o...
  - moffkalast 2y ago
    Actually they did, the entire 15T tokens that were supposedly used for training the llama-3 base models are up on HF as a dataset: https://huggingface.co/datasets/HuggingFaceFW/fineweb https://huggingface.co/datasets/HuggingFaceFW/fineweb It's just not literally labelled so because of obvious reasons.
- TeMPOraL 2y ago
  Redistribute? No. Itemize and link to? Yes. With LLMs, the list doesn't even have to be kept up to date, nor the links alive (though publishing content hashes would go a long way here). It's not like you can get an identical copy of a model built anyway, there's too much randomness at every stage in the process. But, as long as the details of cleanup and training are also open, a list of training material used would suffice - people would fetch parts of it, substitute other parts with equivalents that are open/unlicensed/available, add new sources of their own, and the resulting model should have similar characteristics to the OG one who we could, now, call "open source".
HumanOstrich 2y ago
That is incorrect -- you do not have to provide the full training data to meet the requirements. I recommend reading the actual Open Source AI Definition[1] and the FAQ[2]. There's also the whitepaper[3] that goes into much more detail about the state of affairs. [1]: https://opensource.org/ai/open-source-ai-definition https://opensource.org/ai/open-source-ai-definition [2]: https://hackmd.io/@opensourceinitiative/osaid-faq#What-is-the-role-of-training-data-in-the-Open-Source-AI-Definition https://hackmd.io/@opensourceinitiative/osaid-faq#What-is-th... [3]: https://opensource.org/wp-content/uploads/2025/02/2025-OSI-DataGovernanceOSAI-final-v5.pdf https://opensource.org/wp-content/uploads/2025/02/2025-OSI-D...
- amelius 2y ago
  We need a new definition then.
  - deleted 2y ago
    [deleted]
- Salgat 2y ago
  FYI their open source ai definition released with a lot of controversy, unsurprisingly because it had heavy contribution from corporations with their own interests. It's best to ignore it for now until the wider community has decided on an appropriate open source definition.
deleted 2y ago
[deleted]
2-3-7-43-1807 2y ago
> you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it. the model reveals the architecture which is all you need to use/run/train it.
anfal 2y ago
Fully agreed, someone actually mentioned that to me on reddit and I modified the content + added a disclaimer on top of the article. https://www.reddit.com/r/LocalLLaMA/comments/1ilsfb1/comment/mc2f28e/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button https://www.reddit.com/r/LocalLLaMA/comments/1ilsfb1/comment...

szundi 2y ago

Yes. I cannot comprehend this to this day. A model weights data + runner is how different from a closed source executable? Why do everyone call these open source?

fhd2 2y ago
Because practically speaking, you can fine tune them I suppose? But that's also true for binaries, games are a good example of where people pushed this quite far. Based on what little experience I have in ML, I'd say it's about the same thing. Whereas an API is more akin to a piece of software you can't tinker with in any way. Guess the bar is just lower in the LLM space :P
- darkwater 2y ago
  So a proprietary program with a lot of knobs and configuration files is kind of opensource?
  - fhd2 2y ago
    By what appears to be the logic for "open source AI", a locally executable proprietary program would be "open source" (because you can meddle with the executable). To me, that's mostly just "not SaaS". But somehow, a different definition appears to have stuck for LLMs than for other types of software.
PeterStuer 2y ago
Because typically adapting or improving traditional code to your needs is very difficult without access to the source code and build files. For an LLM you can finetune and enhance, distill and embed given just the model weights, the runtime, and a permissive license. Having more is better. Well written detailed model release papers help a lot. Training code and training data are a great bonus. However, I find the purity contest a bit too dismissive of the great contributions to the AI dev ecosystem that Meta and Deepseek have brought us. Without these, there wouldn't be the open ecosystem we have today.
- GeneralMayhem 2y ago
  It's not a purity contest, it's a clarity contest. If Meta and Deepseek want to operate the way they have been, where they release baked models and whitepapers, that's fine - and you're right, it's certainly more than they're obligated to release. They just shouldn't be calling it "open source" when the source is literally not open.
  - doix 2y ago
    Eh, I can kinda see it. It depends on your definitions of words. People have been muddying the waters with what "open source" means anyway. I have known it to mean code released under an open source license. Other people use it to mean programs where the source is available regardless of license. I would use "source available" to describe that, but some people strongly disagree with my definitions. If I write a program, then obfuscate it and then release the obfuscated code under an open source license, would you consider it open source(I would)? That's kind of the case here, they are releasing the model weights under an open source license. Personally, I think it's fine to shorten it to "open source model" instead of "a model with the weights released under an open source license". What I would object to is releasing model weights under a restrictive license and calling that open source.
    - GeneralMayhem 2y ago
      > If I write a program, then obfuscate it and then release the obfuscated code under an open source license, would you consider it open source(I would)? I wouldn't. Most definitions of open source say something like "in the form used for editing". You can release a built binary under an unrestrictive license, but that does not mean that you've opened the source. It's literally the plain meaning of the words: the source, as in where the thing comes from, needs to be open for it be meaningful.
sigmoid10 2y ago
Open source becomes really complicated once assets with unclear license are involved in any way. Lots of people for example would say that Jedi Knight 2 is open source because Raven Software released the source code and tools needed to build the game. But that alone doesn't mean you can run it, because you still need to get a hold of all the assets (models, textures, sounds) which may or may not still be property of LucasArts or its successors. Even if you have them, it's actually unclear if it is legal to use them this way. So while there are tons of people working on mods and conversions, noone in their right mind would distribute all the source assets. Much in the same way, no sane company will touch the legal nightmare of releasing LLM training data scraped from public websites. Even releasing the LLM alone might be infringement, there are literally court cases being fought over this right now.
- GeneralMayhem 2y ago
  Games like that, or the open-source clones of commercial games that require original assets to play (e.g. OpenXCOM), actually give a very clear analogy here: open source does not mean open assets. The software code is under a separate license from the data it processes. Emulators like Dolphin are kind of in this situation too - the program is open, the data it processes is not. And that's fine! It's still valuable to have access to the source code, even if the "batteries" aren't included. Of course, if you really want to call it an open source model you should include the source for the data scraping/cleaning stages too; then the only thing missing would be the compute time and risk of acquiring dubiously-legal inputs. I personally prefer a taxonomy like: * Open weights: you can download the artifact and run it locally, not just use it through an application like chatgpt or an API. * Open source: the code that created the artifact is provided in the same format that the authors used to work on it. * Open data: the dataset that the source code was used on is available for download. All three of those could be individually licensed or released, for 8 possible combinations. In the analogy to games, they would correspond to the licenses on the retail binary, the source code of the game, and the original uncompressed art assets or Blender projects, respectively.
  - sigmoid10 2y ago
    If it has already been established that open source doesn't mean open assets, why would we change that now? After all, training data is literally nothing but assets - except that you don't need them to run the application. So in that sense open LLMs are more open than these games.
benterix 2y ago
Because Meta called them in this way to differentiate themselves from "Open"AI and everyone else followed the suit.
deleted 2y ago
[deleted]
moffkalast 2y ago
Because marketing (open source is a buzzword after all), and the media just repeats what they read in press releases verbatim. But most people working with the models themselves call them open-weight, except for some occasional exception like OLMo that publishes the dataset and training scripts and is actually open source.
dartos 2y ago
Because Meta called llama that shortly after it got leaked and it stuck. The AI crowd doesn’t care much for licenses anyway.

rettichschnidi 2y ago

Sadly, the OSAID also does not require training data to be available. :(

a2128 2y ago

Is it reasonable to expect companies to redistribute 100TB of copyrighted content they used for their LLM, just on the off-chance someone has a few million laying around and wants to reproduce the model from scratch?

anotherhue 2y ago
The RL-only (no SFT) approaches might remove that issue. Problem sets should be smaller (and mechanically creatable) than the entire western corpus.
puapuapuq 2y ago
For open source model claiming SOTA performance, we could at least check for data leak from its training data.
emseetech 2y ago
Would a reference file with filename, size, source and checksum count towards the OSI definition?
atq2119 2y ago
Perhaps that's not reasonable to expect, but Meta apparently kind of did it anyway, if not in a way that helps reproduce their LLM: https://arstechnica.com/tech-policy/2025/02/meta-torrented-over-81-7tb-of-pirated-books-to-train-ai-authors-say/ https://arstechnica.com/tech-policy/2025/02/meta-torrented-o...
- moffkalast 2y ago
  Actually they did, the entire 15T tokens that were supposedly used for training the llama-3 base models are up on HF as a dataset: https://huggingface.co/datasets/HuggingFaceFW/fineweb https://huggingface.co/datasets/HuggingFaceFW/fineweb It's just not literally labelled so because of obvious reasons.
TeMPOraL 2y ago
Redistribute? No. Itemize and link to? Yes. With LLMs, the list doesn't even have to be kept up to date, nor the links alive (though publishing content hashes would go a long way here). It's not like you can get an identical copy of a model built anyway, there's too much randomness at every stage in the process. But, as long as the details of cleanup and training are also open, a list of training material used would suffice - people would fetch parts of it, substitute other parts with equivalents that are open/unlicensed/available, add new sources of their own, and the resulting model should have similar characteristics to the OG one who we could, now, call "open source".

HumanOstrich 2y ago

That is incorrect -- you do not have to provide the full training data to meet the requirements. I recommend reading the actual Open Source AI Definition[1] and the FAQ[2]. There's also the whitepaper[3] that goes into much more detail about the state of affairs. [1]: https://opensource.org/ai/open-source-ai-definition https://opensource.org/ai/open-source-ai-definition [2]: https://hackmd.io/@opensourceinitiative/osaid-faq#What-is-the-role-of-training-data-in-the-Open-Source-AI-Definition https://hackmd.io/@opensourceinitiative/osaid-faq#What-is-th... [3]: https://opensource.org/wp-content/uploads/2025/02/2025-OSI-DataGovernanceOSAI-final-v5.pdf https://opensource.org/wp-content/uploads/2025/02/2025-OSI-D...

amelius 2y ago
We need a new definition then.
- deleted 2y ago
  [deleted]
Salgat 2y ago
FYI their open source ai definition released with a lot of controversy, unsurprisingly because it had heavy contribution from corporations with their own interests. It's best to ignore it for now until the wider community has decided on an appropriate open source definition.

deleted 2y ago

[deleted]

2-3-7-43-1807 2y ago

> you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it. the model reveals the architecture which is all you need to use/run/train it.

anfal 2y ago

Fully agreed, someone actually mentioned that to me on reddit and I modified the content + added a disclaimer on top of the article. https://www.reddit.com/r/LocalLLaMA/comments/1ilsfb1/comment/mc2f28e/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button https://www.reddit.com/r/LocalLLaMA/comments/1ilsfb1/comment...