8 ms·
> Like, come on, a 14GB of a dump of mainly numbers that were constructed algorithmically are not "source". So if I take a photo of a pretty sunset, release it
by ben_w 3y ago
> Like, come on, a 14GB of a dump of mainly numbers that were constructed algorithmically are not "source".
So if I take a photo of a pretty sunset, release it under MIT license, you'd say it's "not open source" unless I give you the sun and the atmosphere themselves?
These models are perfectly valid things in their own right; the can be fine-tuned or used as parts of other things.
For most of these LLMs (not sure about this one in particular yet) the energy cost in particular of recreation is more than most individuals earn in a lifetime, and the enormous data volume is such that the only people who seriously need this should be copyright lawyers and they should be asking for it to be delivered by station wagon.
- monocasa 3y agoI said "constructed algorithmically". Ie. I expect source to be at the level the engineers who built it generally worked at. It's very nice that they released their build artifacts. It's great that you can take that and make small modifications to it. That doesn't make it open source. > For most of these LLMs (not sure about this one in particular yet) the energy cost in particular of recreation is more than most individuals earn in a lifetime, and the enormous data volume is such that the only people who seriously need this should be copyright lawyers and they should be asking for it to be delivered by station wagon. All of that just sounds like reasons why it's not practical to open source it, not reasons why this release was open source.
- ben_w 3y ago> I said "constructed algorithmically". Ie. I expect source to be at the level the engineers who built it generally worked at. I could either point out that JPEG is an algorithm, or ask if you can recreate a sunset. > All of that just sounds like reasons why it's not practical to open source it No, they're reasons why the stuff you want doesn't matter. If you can actually afford to create a model of your own, you don't need to ask: the entire internet is right there. Some of it even has explicitly friendly licensing terms. An LLM with a friendly license is something you can freely integrate into other things which need friendly licensing. That's valuable all by itself.
- monocasa 3y agoJust because a license is 'friendly' and you don't see the point of an open release, doesn't make it open source. There's been all sorts of closed source libraries that you can freely integrate for whatever reason. They're not open source either.
- zamalek 3y agoYou could train it from scratch on The Pile dataset[1] with a few hundred thousand bucks worth of GPU quota. It's not rocket science - the architecture is, and that's open source by your definition. [1]: https://huggingface.co/datasets/EleutherAI/pile https://huggingface.co/datasets/EleutherAI/pile
- monocasa 3y agoThe graph of layers and ops isn't open source by my definition. It can be extracted from the model, but so can control graphs out of any binary. That's how higher end disassemblers work like IDA and ghidra. Once again, this pickle file is not what's sitting in Mistral's engineer's editors as they go about their day.
- godelski 3y agoWell the checkpoint __is__ the computational graph. The graph is also all the code. But if you want it in python... that's here[0]. Please be clear, we keep asking. What are you asking for? Datasets? Training algo? What? Comparing it to software artifacts isn't a good comparison when any program with open source code (visible or free to use) is equivalent to what's being given. You have everything you need to use, edit, and fuck around with. You don't have the exact scheme, but I'll put it this way, if you gave me the hardware I could produce a LLM of high quality from scratch using their architecture. [0] https://github.com/mistralai/mistral-src/blob/main/mistral/model.py https://github.com/mistralai/mistral-src/blob/main/mistral/m...
- monocasa 3y ago
- lmm 3y ago> So if I take a photo of a pretty sunset, release it under MIT license, you'd say it's "not open source" unless I give you the sun and the atmosphere themselves? You've gotta give me the stuff you used to make it, the stuff you'd want to have if you wanted to recreate a slightly different version of the photo ("in the preferred form for making modifications", as the GPL says). If you just snapped a photo of whatever you saw with whatever camera was in your pocket, then there's nothing else to publish. But if you figured out a timetable of when you should stand where with what kind of lens, then making your photo open-source would mean publishing that timetable. > These models are perfectly valid things in their own right; the can be fine-tuned or used as parts of other things. If the original creator can edit them, and you can't, then that's not open-source; fine-tuning is a help but someone who can only fine-tune is still a second-class user compared to the original developer. The whole point of open source is to put you on an equal footing with the original developer (in particular, to make sure that you can fix bugs by yourself and are never stuck waiting for them to release an update that you need).
- haberman 3y agoI don't think photographs are generally advertised as "open source", because there is not generally a source code form of a photograph. The MIT license doesn't distinguish between binaries and source, but the GPL does. The GPL defines source code as: > The “source code” for a work means the preferred form of the work for making modifications to it. “Object code” means any non-source form of a work. What is the preferred form of an LLM for making modifications to it?
- dragonwriter 3y ago> So if I take a photo of a pretty sunset, release it under MIT license, you'd say it's "not open source" unless I give you the sun and the atmosphere themselves? Phorographs are not source code, are not computer code at all, and do not have a close analog to source code. Calling them “open source” is, at best, a poor and distant metaphor in any case, and so they aren't a useful model for discussing what open source means for software. There's a very reasonable argument that model weights are an IL or object-code like artifact with training data and training source code together as the source code. That doesn't change that the MIT license is an open source license, but when what you release under that license isn’t the whole source aoplicable to the model, but just inference and maybe training code but not the data needed to produce the weights, and final weights, then it is fair to question whether as a whole the model is open source.
- tbrownaw 3y ago> So if I take a photo of a pretty sunset, release it under MIT license, you'd say it's "not open source" unless I give you the sun and the atmosphere themselves? Open source as a concept doesn't really apply to quite a lot of things. Your MIT-licensed photograph is "not open source" in the same way that `{} * {}` is "not a number" (it technically isn't, but that's not quite what NaN is supposed to mean).