5 ms·

Let’s stop using terms like open source falsely. The model isn’t open source, it is open weights. It’s good that the license for the weights is Apache, but for

Let’s stop using terms like open source falsely. The model isn’t open source, it is open weights. It’s good that the license for the weights is Apache, but for this model to be “truly open” they must release training data and source code under an OSI approved license. Otherwise it’s just misleading marketing. So far it seems like Snowflake will release some blog posts and “cookbooks”, whatever that means, but not actual training source code. Only the inference code is open source here, which is uninteresting.

deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
jerrygenser 3y ago
> they must release training data and source code under an OSI approved license The source code is also apache 2.0
- blackeyeblitzar 3y ago
  Snowflake has only released the inference code - meaning the code you need to "run" the model. So if you take the weights they have released (which is the model that is a result of training), you can host the weights and inference code, and feed prompts to it, to get answers. But you don't have the actual source code you need to produce the weights in the first place. As an example of what open source actually means for LLMs, you can look at what AI2 does with their OLMo model (https://allenai.org/olmo https://allenai.org/olmo), where each model that they release comes with: > Full training data used for these models, including code that produces the training data, from AI2’s Dolma, and WIMBD for analyzing pretraining data. > Full model weights, training code, training logs, training metrics in the form of Weights & Biases logs, and inference code. > 500+ checkpoints per model, from every 1000 steps during the training process, available as revisions on HuggingFace. > Evaluation code under the umbrella of AI2’s Catwalk and Paloma. > Fine-tuning code and adapted models (with Open Instruct) > All code, weights, and intermediate checkpoints are released under the Apache 2.0 License. OLMo is what "truly open" is, while the rest is openwashing and marketing.
WhitneyLand 3y ago
What’s the problem? This is what it says on their repo home page. ————— Truly Open: Apache 2.0 license provides ungated access to weights and code. In addition, we are also open sourcing all of our data recipes and research insights.
- blackeyeblitzar 3y ago
  The source code they're talking about is not the training code. The only thing I saw released was their inference code and weights. You can verify this by visiting the following: https://github.com/Snowflake-Labs/snowflake-arctic/tree/main https://github.com/Snowflake-Labs/snowflake-arctic/tree/main https://huggingface.co/Snowflake/snowflake-arctic-base https://huggingface.co/Snowflake/snowflake-arctic-base https://huggingface.co/Snowflake/snowflake-arctic-instruct https://huggingface.co/Snowflake/snowflake-arctic-instruct To put it another way, when they share the weights for the model, that's like sharing the compiled output for some software - like releasing an executable instead of the source code that can produce the executable. They aren't sharing the things you need to produce the weights (the training code, training data, any preprocessing code, etc). Without those inputs you actually cannot even audit or verify how the model works. The team making the model might bias the model in all sorts of ways without your knowledge.
ko27 3y ago
You have a weird definition of open source. OS software developers don't release the books they have read or the tools they've used to write code. This is fully 100% OSI compliant source code with an approved license (Apache 2.0). You are not entitled to anything more than this.
- Zambyte 3y ago
  They don't have a weird definition of open source. I recently outlined a LLM chat that I think clearly outlines this: https://news.ycombinator.com/item?id=40035688 https://news.ycombinator.com/item?id=40035688
  - ko27 3y ago
    A bunch of code was autocompleted or generated by IDEs, are open source developers supposed to release the source code of that IDE to be OSI compliant?
    - Zambyte 3y ago
      Is the IDE a primary input for building the program? Is the IDE a build dependency? Probably not. Certainly not based on the situation you described. The LLM equivalent here would be programmatically generating synthetic input or cleaning input for training. You don't need the tools used to generate or clean the data in order to train the model, and thus they can be propriety in the context of an open source model, so long as the source for the model is open (the training data).
      - ko27 3y ago
        > Is the IDE a primary input for building the program? Is the IDE a build dependency? No, the same way training is not a build dependency for the weights source code. You can literally compile and run them without any training data.
        Zambyte 3y ago
dantheman 3y ago
Why would they need to release the training data? that's nonsense.
- Zambyte 3y ago
  Because the training data is the source of the model. This thread may illuminate it for you: https://news.ycombinator.com/item?id=40035688 https://news.ycombinator.com/item?id=40035688 Most models that are described as "open source" are actually open weight, because their source is not open.
  - dantheman 3y ago
    It's still open source and can be used; Just like open source refers to code, not all the design documents, discussion, plans, etc.
- blackeyeblitzar 3y ago
  Open source for traditional software means that you can see how the software works and reproduce the executable by compiling the software from source code. For LLMs, reproducing the model means reproducing the weights. And to do that you need the training source code AND the training data. There are already other great models that do this (see my comment at https://news.ycombinator.com/item?id=40147298 https://news.ycombinator.com/item?id=40147298). I get that there may be some training data that is proprietary and cannot be released. But in those scenarios, it would still be good to know what the data is, how it was curated or filtered (this greatly affects LLM performance), how it is weighted relative to other training data, and so forth. But a significant portion of data used to train models is not proprietary and in those cases they can simply link to that data elsewhere or release it themselves, which is what others have done.
  - furyofantares 3y ago
    There's no perfect analogy. It's far easier to usefully modify the weights of a model without the training data than it is to modify a binary executable without its source code. I'd rather also have the data for sure! But in terms of what useful things I can do with it, weights are closer to source code than they are to a binary blob.
cqqxo4zV46cp 3y ago
Thankfully, thankfully, this sort of stuff isn’t decided based on the personal reckoning of someone on Hacker News. Whether or not training data needs to be open source in order for the resulting model to be open source is, at the very least, up for debate. And that’s a charitable interpretation. This is quite clearly instead your view based on your own personal philosophy. Software licenses are legal instruments, not a vague notion of some ideology. If you don’t think that the model is open source, you’ve obviously seen legal precedent that nobody else has.
- stefan_ 3y ago
  What? You know the people writing open source licenses have spent more than 5 minutes thinking about this, right? The GPL says it straight up: > The “source code” for a work means the preferred form of the work for making modifications to it Clearly just weights don't qualify, just like C run through an obfuscator would not count.
- Zambyte 3y ago
  The training data is the source. If the training data is not open, the model is not open source, because the source of the model is not open. See this previous comment of mine that explains this: https://news.ycombinator.com/item?id=40035688 https://news.ycombinator.com/item?id=40035688
deleted 3y ago
[deleted]
jeffra45 3y ago
By truly open, we mean our releases use an OSI-recognized license (Apache-2) and we go beyond just model weights. Here are the things that we are open-sourcing: i) Open-Sourced Model Weights ii) Open-Sourced Fine-Tuning Pipeline. This is essentially the training code if you want to adapt this model to your use cases. This along with an associated cookbook will be released soon, so keep an eye on our repo for updates: https://github.com/Snowflake-Labs/snowflake-arctic/ https://github.com/Snowflake-Labs/snowflake-arctic/ iii) Open-Sourced Data Information: We trained on publicly available datasets, and we will share information on what these datasets are, how we processed and filtered them, composition of our datasets etc. They will be published as part of the cookbook series here: https://www.snowflake.com/en/data-cloud/arctic/cookbook/ https://www.snowflake.com/en/data-cloud/arctic/cookbook/, shortly. iv) Open-Sourced Research: We will share all of our findings from our architecture studies, performance analysis etc. Again these will be published as part of the cookbook series. You can already see a few blogs covering MoE Architecture and Training Systems here: https://medium.com/snowflake/snowflake-arctic-cookbook-series-exploring-mixture-of-experts-moe-c7d6b8f14d16 https://medium.com/snowflake/snowflake-arctic-cookbook-serie..., https://medium.com/snowflake/snowflake-arctic-cookbook-series-building-an-efficient-training-system-for-arctic-6658b9bdfcae https://medium.com/snowflake/snowflake-arctic-cookbook-serie... v) Pre-Training System information: We actually used the already open-sourced libraries DeepSpeed and Megatron-DeepSpeed for training optimizations and the model implementation for training the model. We have already upstreamed several improvements and fixes to these libraries and will continue to do so. Our cookbooks provide the necessary information on the architecture and system configurations.
- sroussey 3y ago
  It would be awesome if things weren’t rushed such that you have to say “we will” so often, rather than “here is the link”. It’s awesome the work you all have done. But not sure if I’ll return and remember the “we will” stuff, meaning that I’m not likely every look at it or start using it.

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

jerrygenser 3y ago

> they must release training data and source code under an OSI approved license The source code is also apache 2.0

blackeyeblitzar 3y ago
Snowflake has only released the inference code - meaning the code you need to "run" the model. So if you take the weights they have released (which is the model that is a result of training), you can host the weights and inference code, and feed prompts to it, to get answers. But you don't have the actual source code you need to produce the weights in the first place. As an example of what open source actually means for LLMs, you can look at what AI2 does with their OLMo model (https://allenai.org/olmo https://allenai.org/olmo), where each model that they release comes with: > Full training data used for these models, including code that produces the training data, from AI2’s Dolma, and WIMBD for analyzing pretraining data. > Full model weights, training code, training logs, training metrics in the form of Weights & Biases logs, and inference code. > 500+ checkpoints per model, from every 1000 steps during the training process, available as revisions on HuggingFace. > Evaluation code under the umbrella of AI2’s Catwalk and Paloma. > Fine-tuning code and adapted models (with Open Instruct) > All code, weights, and intermediate checkpoints are released under the Apache 2.0 License. OLMo is what "truly open" is, while the rest is openwashing and marketing.

WhitneyLand 3y ago

What’s the problem? This is what it says on their repo home page. ————— Truly Open: Apache 2.0 license provides ungated access to weights and code. In addition, we are also open sourcing all of our data recipes and research insights.

blackeyeblitzar 3y ago
The source code they're talking about is not the training code. The only thing I saw released was their inference code and weights. You can verify this by visiting the following: https://github.com/Snowflake-Labs/snowflake-arctic/tree/main https://github.com/Snowflake-Labs/snowflake-arctic/tree/main https://huggingface.co/Snowflake/snowflake-arctic-base https://huggingface.co/Snowflake/snowflake-arctic-base https://huggingface.co/Snowflake/snowflake-arctic-instruct https://huggingface.co/Snowflake/snowflake-arctic-instruct To put it another way, when they share the weights for the model, that's like sharing the compiled output for some software - like releasing an executable instead of the source code that can produce the executable. They aren't sharing the things you need to produce the weights (the training code, training data, any preprocessing code, etc). Without those inputs you actually cannot even audit or verify how the model works. The team making the model might bias the model in all sorts of ways without your knowledge.

ko27 3y ago

You have a weird definition of open source. OS software developers don't release the books they have read or the tools they've used to write code. This is fully 100% OSI compliant source code with an approved license (Apache 2.0). You are not entitled to anything more than this.

Zambyte 3y ago
They don't have a weird definition of open source. I recently outlined a LLM chat that I think clearly outlines this: https://news.ycombinator.com/item?id=40035688 https://news.ycombinator.com/item?id=40035688
- ko27 3y ago
  A bunch of code was autocompleted or generated by IDEs, are open source developers supposed to release the source code of that IDE to be OSI compliant?
  - Zambyte 3y ago
    Is the IDE a primary input for building the program? Is the IDE a build dependency? Probably not. Certainly not based on the situation you described. The LLM equivalent here would be programmatically generating synthetic input or cleaning input for training. You don't need the tools used to generate or clean the data in order to train the model, and thus they can be propriety in the context of an open source model, so long as the source for the model is open (the training data).
    - ko27 3y ago
      > Is the IDE a primary input for building the program? Is the IDE a build dependency? No, the same way training is not a build dependency for the weights source code. You can literally compile and run them without any training data.
      - Zambyte 3y ago

dantheman 3y ago

Why would they need to release the training data? that's nonsense.

Zambyte 3y ago
Because the training data is the source of the model. This thread may illuminate it for you: https://news.ycombinator.com/item?id=40035688 https://news.ycombinator.com/item?id=40035688 Most models that are described as "open source" are actually open weight, because their source is not open.
- dantheman 3y ago
  It's still open source and can be used; Just like open source refers to code, not all the design documents, discussion, plans, etc.
blackeyeblitzar 3y ago
Open source for traditional software means that you can see how the software works and reproduce the executable by compiling the software from source code. For LLMs, reproducing the model means reproducing the weights. And to do that you need the training source code AND the training data. There are already other great models that do this (see my comment at https://news.ycombinator.com/item?id=40147298 https://news.ycombinator.com/item?id=40147298). I get that there may be some training data that is proprietary and cannot be released. But in those scenarios, it would still be good to know what the data is, how it was curated or filtered (this greatly affects LLM performance), how it is weighted relative to other training data, and so forth. But a significant portion of data used to train models is not proprietary and in those cases they can simply link to that data elsewhere or release it themselves, which is what others have done.
- furyofantares 3y ago
  There's no perfect analogy. It's far easier to usefully modify the weights of a model without the training data than it is to modify a binary executable without its source code. I'd rather also have the data for sure! But in terms of what useful things I can do with it, weights are closer to source code than they are to a binary blob.

cqqxo4zV46cp 3y ago

Thankfully, thankfully, this sort of stuff isn’t decided based on the personal reckoning of someone on Hacker News. Whether or not training data needs to be open source in order for the resulting model to be open source is, at the very least, up for debate. And that’s a charitable interpretation. This is quite clearly instead your view based on your own personal philosophy. Software licenses are legal instruments, not a vague notion of some ideology. If you don’t think that the model is open source, you’ve obviously seen legal precedent that nobody else has.

stefan_ 3y ago
What? You know the people writing open source licenses have spent more than 5 minutes thinking about this, right? The GPL says it straight up: > The “source code” for a work means the preferred form of the work for making modifications to it Clearly just weights don't qualify, just like C run through an obfuscator would not count.
Zambyte 3y ago
The training data is the source. If the training data is not open, the model is not open source, because the source of the model is not open. See this previous comment of mine that explains this: https://news.ycombinator.com/item?id=40035688 https://news.ycombinator.com/item?id=40035688

deleted 3y ago

[deleted]

jeffra45 3y ago

By truly open, we mean our releases use an OSI-recognized license (Apache-2) and we go beyond just model weights. Here are the things that we are open-sourcing: i) Open-Sourced Model Weights ii) Open-Sourced Fine-Tuning Pipeline. This is essentially the training code if you want to adapt this model to your use cases. This along with an associated cookbook will be released soon, so keep an eye on our repo for updates: https://github.com/Snowflake-Labs/snowflake-arctic/ https://github.com/Snowflake-Labs/snowflake-arctic/ iii) Open-Sourced Data Information: We trained on publicly available datasets, and we will share information on what these datasets are, how we processed and filtered them, composition of our datasets etc. They will be published as part of the cookbook series here: https://www.snowflake.com/en/data-cloud/arctic/cookbook/ https://www.snowflake.com/en/data-cloud/arctic/cookbook/, shortly. iv) Open-Sourced Research: We will share all of our findings from our architecture studies, performance analysis etc. Again these will be published as part of the cookbook series. You can already see a few blogs covering MoE Architecture and Training Systems here: https://medium.com/snowflake/snowflake-arctic-cookbook-series-exploring-mixture-of-experts-moe-c7d6b8f14d16 https://medium.com/snowflake/snowflake-arctic-cookbook-serie..., https://medium.com/snowflake/snowflake-arctic-cookbook-series-building-an-efficient-training-system-for-arctic-6658b9bdfcae https://medium.com/snowflake/snowflake-arctic-cookbook-serie... v) Pre-Training System information: We actually used the already open-sourced libraries DeepSpeed and Megatron-DeepSpeed for training optimizations and the model implementation for training the model. We have already upstreamed several improvements and fixes to these libraries and will continue to do so. Our cookbooks provide the necessary information on the architecture and system configurations.

sroussey 3y ago
It would be awesome if things weren’t rushed such that you have to say “we will” so often, rather than “here is the link”. It’s awesome the work you all have done. But not sure if I’ll return and remember the “we will” stuff, meaning that I’m not likely every look at it or start using it.