4 ms·
>> tokenizer.model in their release is absolutely a binary Is this not a BPE+sentencepiece? It is quite usual practice when you do the training or even prepare
by pk-protect-ai 3y ago
>> tokenizer.model in their release is absolutely a binary
Is this not a BPE+sentencepiece? It is quite usual practice when you do the training or even prepare the data with fairseq ...
EDIT: I mean it will be a binary file for the tokenizer model after all but I see no problem here ...
- monocasa 3y agoJust because you don't see a problem with it being a binary, doesn't make it source. It's still a build artifact.
- pk-protect-ai 3y ago[flagged]
- monocasa 3y agoThat's one of the libraries, not the source of this binary release. What does Mistral's engineers edit before sending this model off to be trained? That's the source.
- pk-protect-ai 3y agoIt is literally takes 20 min on my PC to prepare multilanguage corpus and train BPE+sentencepiece tokenizer with fairseq. You have all the documentation in there. If you do not know how to use these tools, does not mean they are not there. You literally do not need to edit anything.
- monocasa 3y agoSweet. I've also been able to reverse engineer stripped binaries and reimplement them; I talked at a major conference last year about that actually. That doesn't make this release anymore source.