5 ms·
> these releases of model binaries Are they binaries? I haven't seen a binary in awhile tbh. Usually they're releasing both the raw architecture (i.e. code) an
by godelski 3y ago
> these releases of model binaries
Are they binaries? I haven't seen a binary in awhile tbh. Usually they're releasing both the raw architecture (i.e. code) and the weights of the models (i.e. what numbers go into what parts of the architecture). The latter is in a readable format that you can generally edit by hand if you wanted to. But even if it was a binary as long as you have the architecture you can always load into the model and decide if you want to probe it (extract values) or modify it by tuning (many methods to do this).
As far as I'm concerned, realistically the only issue here is the standard issue around the open source definition. Does it mean the source is open as available or open as "do what the fuck you want"? I mean it's not like OpenAI is claiming that GPT is open sourced. It's just that Meta did and their source is definitely visible. Fwiw, they are the only major company to do so. Google doesn't open source: they, like OpenAI, use private datasets and private models. I'm more upset at __Open__AI and Google than I am about Meta. To me people are barking up the wrong tree here. (It also feels weird that Meta is the "good guy" here... relatively at least)
Edit: I downloaded their checkpoint. It is the standard "pth" file. This is perfectly readable, it is just a pickle file. I like to use graftr to view checkpoints, but other tools exist (https://github.com/lmnt-com/graftr https://github.com/lmnt-com/graftr)
- monocasa 3y agoFirst off tokenizer.model in their release is absolutely a binary by any definition. Secondly, is hand editing the pickle file the way Mistral's engineers constructed this pickle file? Why not? The automation that constructed that file (and tokenizer.model) is the source. Binaries in this context can certainly be an ASCII encoded file. Just because other vendors are worse doesn't make this open source.
- syntaxing 3y agoNot OP but I don’t get it though, you can modify the tokenizer all you want and fine tune the weights all you want. There’s nothing inherently hidden behind a binary
- monocasa 3y agoI can edit binaries too. The question is am I provided the build source that constructed these files. Mistral did not hand edit these files to construct them, there's source out there that built them. Like, come on, a 14GB of a dump of mainly numbers that were constructed algorithmically are not "source".
- syntaxing 3y agoIf I’m understanding you correctly, what you mean is that’s it’s only truly open source if they provide the data they used to train it as well?
- monocasa 3y agoIf that's what's needed to work at the level their engineers work on the model. Which is true of traditional software as well. You don't get to call your binary open source just because you have licensed materials in there you can't release.
- lawlessone 3y agoIs a database software only open source if they release with data?
- monocasa 3y agoIs the data what the database engineers edit and add to their build pipeline in order to build the database software?
- cafxx 3y agoNot to be the devil's advocate here, but almost certainly it can be the case that data was used to define heuristics (potentially using automated statistical methods) that a engineer then formalized as code. Without that data that specific heuristic wouldn't exist, at least very likely not in that form. Yet that data does not have to be included in any open source release. And obviously you as a recipient of the release can modify the heuristic (or at least, you can modify the version that was codified), but you can not reconstruct it from the original data. I know my example is not exactly what is happening here, but the two sound pretty affine to me and there seem to be a fairly blurry line dividing the two... so I would argue that where "this must be included in a open source release" ends and "this does not need to be included in a open source release" starts is not always so cut and dry. (A variant of this, that happens fairly frequently, is when you find a commit that says something along the lines of "this change was made because it made an internal, non-public workload X% faster"; if the data that measurement is based upon did not exist, or if the workload itself didn't exist, that change wouldn't have been made, or maybe it would have been made differently... so again you end up with logic due to data that is not in the open source release) If we want to go one step further, we could even ask: what about static assets (e.g. images, photographs, other datasets, etc.) included in a open-source release... maybe I'm dead wrong here, but I have never heard that such assets must themselves be "reproducible from source" (what even is, in this context, the "source" of a photograph?). That being said, I sure wish the training data used for all of these models was available to everyone...
- pk-protect-ai 3y ago>> tokenizer.model in their release is absolutely a binary Is this not a BPE+sentencepiece? It is quite usual practice when you do the training or even prepare the data with fairseq ... EDIT: I mean it will be a binary file for the tokenizer model after all but I see no problem here ...
- monocasa 3y agoJust because you don't see a problem with it being a binary, doesn't make it source. It's still a build artifact.
- pk-protect-ai 3y ago[flagged]
- monocasa 3y agoThat's one of the libraries, not the source of this binary release. What does Mistral's engineers edit before sending this model off to be trained? That's the source.
- pk-protect-ai 3y agoIt is literally takes 20 min on my PC to prepare multilanguage corpus and train BPE+sentencepiece tokenizer with fairseq. You have all the documentation in there. If you do not know how to use these tools, does not mean they are not there. You literally do not need to edit anything.
- monocasa 3y agoSweet. I've also been able to reverse engineer stripped binaries and reimplement them; I talked at a major conference last year about that actually. That doesn't make this release anymore source.
- WanderPanda 3y ago„the automation“ is probably manual and not even deterministic
- monocasa 3y agoI guarantee you there's automation around training this model. There's also the factor of the dataset itself. And it doesn't matter much if it's perfectly deterministic. Source builds of traditional software aren't typically fully reproducible either. That doesn't change And I give you better than coin flip odds that it is actually deterministic. The engineers at the big ML shops I've had conversations with have been doing deterministic training for quite some; they believed it was key to training at scale. That's what gives you the "did this model go way off the deep end because of something we did in the model, or because a training GPU is on the fritz".
- GuB-42 3y ago> The “source code” for a work means the preferred form of the work for making modifications to it. That's the definition in the GPL. That it is text or binary doesn't matter. So are the weights the preferred form for making modifications? Partly yes, because of fine tuning, but also no, because you are limited in what you can do with fine tuning. If Mistral had to make major changes to their model, they would probably start with the dataset and code they have but you don't, the one that created the weights file. So I wouldn't call it "open source", just "open". You can do whatever you want with what you have, but you don't have the same abilities as Mistral to modify the model because you lack some data. Still, it is a bit of an unusual situation since even with the "real sources", i.e. training data and code, most people wouldn't have the resources to retrain the model, and a big part of the value in these models is the computing resources that were invested in training them.