3 ms·

Hey, tone down the agression. Programs aren't the only things that can be open-source. LLMs are made of more than just programs. Some of those things that are

Hey, tone down the agression. Programs aren't the only things that can be open-source. LLMs are made of more than just programs. Some of those things that are not programs are not open source. If any part of something is not open source, the whole of that thing is not open source, per definition. Therefor any LLM that has any parts that are not open source is not open source, even if some parts of it are open source.

darksaints 2y ago
That simply isn't true. LLM weights are an output of a model training process. They are also an input of a model inference process. Providing weights for people to use does not change the source code in any way, shape, or form. While a model does require weights in order to function, there is nothing about the model that requires you to use any weights provided by anybody, regardless of how they were trained. The model is open source. You can train a Llama 3.1 model from scratch on your proprietary collection of alien tentacle erotica, and you can do so precisely because the model is open source. You can claim that the weights themselves are not open source, but that says absolutely nothing about the model being open source. The weights distributed are simply not required. But more importantly, under your definition, there will never exist in any form a useful open source set of weights. Because almost all data is proprietary. Anybody can train on large quantities of proprietary data without permission using fair use protections, but no matter what you can't redistribute it without permission. Any weights derived from training a model on data that can be redistributed by a single entity would inherently be so tiny that it would be almost useless. You could create a model with a few billion parameters that could memorize it all verbatim. Open weights can be useful, and they can be a huge boon to users that don't have the resources to train large models, but they aren't required for any meaningful definition of open source.
- guerrilla 2y ago
  > But more importantly, under your definition, there will never exist in any form a useful open source set of weights. Because almost all data is proprietary. Anybody can train on large quantities of proprietary data without permission using fair use protections, but no matter what you can't redistribute it without permission. Any weights derived from training a model on data that can be redistributed by a single entity would inherently be so tiny that it would be almost useless. You could create a model with a few billion parameters that could memorize it all verbatim. That may very well be so. We'll see what the future holds for us.
jncfhnb 2y ago
The training data to create an LLM are as much a part of the LLM as the design notes and IDE use to create traditional software are a part of those projects. They’re not required for it to be open sourced.

darksaints 2y ago

That simply isn't true. LLM weights are an output of a model training process. They are also an input of a model inference process. Providing weights for people to use does not change the source code in any way, shape, or form. While a model does require weights in order to function, there is nothing about the model that requires you to use any weights provided by anybody, regardless of how they were trained. The model is open source. You can train a Llama 3.1 model from scratch on your proprietary collection of alien tentacle erotica, and you can do so precisely because the model is open source. You can claim that the weights themselves are not open source, but that says absolutely nothing about the model being open source. The weights distributed are simply not required. But more importantly, under your definition, there will never exist in any form a useful open source set of weights. Because almost all data is proprietary. Anybody can train on large quantities of proprietary data without permission using fair use protections, but no matter what you can't redistribute it without permission. Any weights derived from training a model on data that can be redistributed by a single entity would inherently be so tiny that it would be almost useless. You could create a model with a few billion parameters that could memorize it all verbatim. Open weights can be useful, and they can be a huge boon to users that don't have the resources to train large models, but they aren't required for any meaningful definition of open source.

guerrilla 2y ago
> But more importantly, under your definition, there will never exist in any form a useful open source set of weights. Because almost all data is proprietary. Anybody can train on large quantities of proprietary data without permission using fair use protections, but no matter what you can't redistribute it without permission. Any weights derived from training a model on data that can be redistributed by a single entity would inherently be so tiny that it would be almost useless. You could create a model with a few billion parameters that could memorize it all verbatim. That may very well be so. We'll see what the future holds for us.

jncfhnb 2y ago

The training data to create an LLM are as much a part of the LLM as the design notes and IDE use to create traditional software are a part of those projects. They’re not required for it to be open sourced.