6 ms·
"Open weights" is a more appropriate term but I'll point out that these weights are also largely inscrutable to the people with the code that trained it. And fo
by causal 2y ago
"Open weights" is a more appropriate term but I'll point out that these weights are also largely inscrutable to the people with the code that trained it. And for licensing reasons, the datasets may not be possible to share.
There is still a lot of modifying you can do with a set of weights, and they make great foundations for new stuff, but yeah we may never see a competitive model that's 100% buildable at home.
Edit: mkolodny points out that the model code is shared (under llama license at least), which is really all you need to run training https://github.com/meta-llama/llama3/blob/main/llama/model.py https://github.com/meta-llama/llama3/blob/main/llama/model.p...
- ajxlasA 2y agoReally? I have to check out the training code again. Last time I looked the training and inference code were just example toys that were barely usable. Has that changed?
- deleted 2y ago[deleted]
- aerzen 2y agoLLAMA is an open-weights model. I like this term, let's use that instead of open source.
- gowld 2y agoCan a human programmer edit the weights according to some semantics?
- sebastiennight 2y agoIt is possible to merge two fine-tunes of models from the same family by... wait for it... averaging or combining their weights[0]. I am still amazed that we can do that. [0]: https://arxiv.org/abs/2212.09849 https://arxiv.org/abs/2212.09849
- the8thbit 2y agoThis is absolutely wild.
- root_axis 2y agoYes. Using fine tuning.
- sitkack 2y agoYes, there is the concept of a "frakenmerge" and folks have also bolted on vision and audio models to LLMs.
- stavros 2y ago"Open weights" means you can use the weights for free (as in beer). "Open source" means you get the training dataset and the methodology. ~Nobody does open source LLMs.
- _heimdall 2y agoWhy is the dataset required for it to be open source? If I self host a project that is open sourced rather than paying for a hosted version, like Sentry.io for example, I don't expect data to come along with the code. Licensing rights are always up for debate in open source, but I wouldn't expect more than the code to be available and reviewable for anything needed to build and run the project. In the case of an LLM I would expect that to mean the code run to train the model, the code for the model data structure itself, and the control code for querying the model should all be available. I'm not actually sure if Meta does share all that, but training data is separate from open source IMO.
- solarmist 2y agoThe sticking point is you can’t build the model. To be able to build the model from scratch you need methodology and a complete description of the data set. They only give you a blob of data you can run.
- _heimdall 2y agoGot it, that makes sense. I still wouldn't expect them to have to publicly share the data itself, but if you can't take the code they share and run it against your own data to build a model that wouldn't be open source in my understanding of it.
- TeMPOraL 2y agoData is the source code here, though. Training code is effectively a build script. Data that goes into training a model does not function like assets in videogames; you can't swap out the training dataset after release and get substantially the same thing. If anything, you can imagine the weights themselves are the asset - and even if the vendor is granting most users a license to copy and modify it (unlike with videogames), the asset itself isn't open source. So, the only bit that's actually open-sourced in these models is the inference code. But that's a trivial part that people can procure equivalents of elsewhere or reproduce from published papers. In this sense, even if you think calling the models "open source" is correct, it doesn't really mean much, because the only parts that matter are not open sourced.
- ab5tract 2y agoIf you can’t share the dataset, under what twisted reality are you fine to share the derivative models based on those unsharable datasets? In a better world, there would be no “I ran some algos on it and now it’s mine” defense.
- guitarlimeo 2y agoYeah was gonna say exactly the same thing. Weird how the legislation allows releasing LLMs trained on data that is not allowed to be shared otherwise.
- floydnoel 2y agoMeta might possibly have a license to use (some of) that data, but not a license to distribute it. Legislation has little to do with it, I imagine.
- yangcheng 2y agolatest llama 3.1 is in a different repo, https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/api/model.py https://github.com/meta-llama/llama-models/blob/main/models/... , but yes, the code is shared. It astonishing that in software 2.0 era, powerful applications like llama has only hundreds of lines of code, and most work hidden in training data. Source code alone is no longer that informative as Software 1.0
- twelvechairs 2y agoOpen training data would be great too. If you have open data and open source code you can reproduce the weights
- blharr 2y agoNot easily for these large scale models, but theoretically maybe
- danielrhodes 2y agoFor models of this size, the code used to train them is going to be very custom to the architecture/cluster they are built on. It would be almost useless to anybody outside of Meta. The dataset would be more a lot more interesting, as it would at the very least show everybody how they got it to behave in certain ways.