3 ms·

I said "constructed algorithmically". Ie. I expect source to be at the level the engineers who built it generally worked at. It's very nice that they released

I said "constructed algorithmically". Ie. I expect source to be at the level the engineers who built it generally worked at. It's very nice that they released their build artifacts. It's great that you can take that and make small modifications to it. That doesn't make it open source. > For most of these LLMs (not sure about this one in particular yet) the energy cost in particular of recreation is more than most individuals earn in a lifetime, and the enormous data volume is such that the only people who seriously need this should be copyright lawyers and they should be asking for it to be delivered by station wagon. All of that just sounds like reasons why it's not practical to open source it, not reasons why this release was open source.

ben_w 3y ago
> I said "constructed algorithmically". Ie. I expect source to be at the level the engineers who built it generally worked at. I could either point out that JPEG is an algorithm, or ask if you can recreate a sunset. > All of that just sounds like reasons why it's not practical to open source it No, they're reasons why the stuff you want doesn't matter. If you can actually afford to create a model of your own, you don't need to ask: the entire internet is right there. Some of it even has explicitly friendly licensing terms. An LLM with a friendly license is something you can freely integrate into other things which need friendly licensing. That's valuable all by itself.
- monocasa 3y ago
  Just because a license is 'friendly' and you don't see the point of an open release, doesn't make it open source. There's been all sorts of closed source libraries that you can freely integrate for whatever reason. They're not open source either.
  - zamalek 3y ago
    You could train it from scratch on The Pile dataset[1] with a few hundred thousand bucks worth of GPU quota. It's not rocket science - the architecture is, and that's open source by your definition. [1]: https://huggingface.co/datasets/EleutherAI/pile https://huggingface.co/datasets/EleutherAI/pile
    - monocasa 3y ago
      The graph of layers and ops isn't open source by my definition. It can be extracted from the model, but so can control graphs out of any binary. That's how higher end disassemblers work like IDA and ghidra. Once again, this pickle file is not what's sitting in Mistral's engineer's editors as they go about their day.
      - godelski 3y ago
        Well the checkpoint __is__ the computational graph. The graph is also all the code. But if you want it in python... that's here[0]. Please be clear, we keep asking. What are you asking for? Datasets? Training algo? What? Comparing it to software artifacts isn't a good comparison when any program with open source code (visible or free to use) is equivalent to what's being given. You have everything you need to use, edit, and fuck around with. You don't have the exact scheme, but I'll put it this way, if you gave me the hardware I could produce a LLM of high quality from scratch using their architecture. [0] https://github.com/mistralai/mistral-src/blob/main/mistral/model.py https://github.com/mistralai/mistral-src/blob/main/mistral/m...
        monocasa 3y ago
        That doesn't conflict with anything I've said. Yes, the checkpoint is code. It's not source code. It's not what Mistral's engineers edit to create this release. Just like an ELF file is necessarily contains the code flow graph, in a way extractable by experts, but isn't open source because... it's not source.
        zamalek 3y ago
        In that analogy the source of the ELF would be hand-written assembly. There aren't many layers of abstraction above the graph when it comes to ML.
- dartos 3y ago
  The permissiveness of the license with regards to use isn’t the crux of the argument. The open source family of licenses are about freedom. If I’m not given the tools to recreate a model, then I’m not afforded the freedoms normally associated with these open licenses. Really there’s little difference between Apache and CC-BY here.

ben_w 3y ago

> I said "constructed algorithmically". Ie. I expect source to be at the level the engineers who built it generally worked at. I could either point out that JPEG is an algorithm, or ask if you can recreate a sunset. > All of that just sounds like reasons why it's not practical to open source it No, they're reasons why the stuff you want doesn't matter. If you can actually afford to create a model of your own, you don't need to ask: the entire internet is right there. Some of it even has explicitly friendly licensing terms. An LLM with a friendly license is something you can freely integrate into other things which need friendly licensing. That's valuable all by itself.

monocasa 3y ago
Just because a license is 'friendly' and you don't see the point of an open release, doesn't make it open source. There's been all sorts of closed source libraries that you can freely integrate for whatever reason. They're not open source either.
- zamalek 3y ago
  You could train it from scratch on The Pile dataset[1] with a few hundred thousand bucks worth of GPU quota. It's not rocket science - the architecture is, and that's open source by your definition. [1]: https://huggingface.co/datasets/EleutherAI/pile https://huggingface.co/datasets/EleutherAI/pile
  - monocasa 3y ago
    The graph of layers and ops isn't open source by my definition. It can be extracted from the model, but so can control graphs out of any binary. That's how higher end disassemblers work like IDA and ghidra. Once again, this pickle file is not what's sitting in Mistral's engineer's editors as they go about their day.
    - godelski 3y ago
      Well the checkpoint __is__ the computational graph. The graph is also all the code. But if you want it in python... that's here[0]. Please be clear, we keep asking. What are you asking for? Datasets? Training algo? What? Comparing it to software artifacts isn't a good comparison when any program with open source code (visible or free to use) is equivalent to what's being given. You have everything you need to use, edit, and fuck around with. You don't have the exact scheme, but I'll put it this way, if you gave me the hardware I could produce a LLM of high quality from scratch using their architecture. [0] https://github.com/mistralai/mistral-src/blob/main/mistral/model.py https://github.com/mistralai/mistral-src/blob/main/mistral/m...
      - monocasa 3y ago
        That doesn't conflict with anything I've said. Yes, the checkpoint is code. It's not source code. It's not what Mistral's engineers edit to create this release. Just like an ELF file is necessarily contains the code flow graph, in a way extractable by experts, but isn't open source because... it's not source.
        zamalek 3y ago
        In that analogy the source of the ELF would be hand-written assembly. There aren't many layers of abstraction above the graph when it comes to ML.
dartos 3y ago
The permissiveness of the license with regards to use isn’t the crux of the argument. The open source family of licenses are about freedom. If I’m not given the tools to recreate a model, then I’m not afforded the freedoms normally associated with these open licenses. Really there’s little difference between Apache and CC-BY here.