5 ms·

The whole point of machine learning is deriving an algorithm from data. This is the algorithm they derived. It's open source. You can use it or change it. Havin

The whole point of machine learning is deriving an algorithm from data. This is the algorithm they derived. It's open source. You can use it or change it. Having the data that was used to derive it is not relevant.

monocasa 3y ago
> It's open source. How did the engineers who built it do so? Is there more source to create this build artifact?
- syntaxing 3y ago
  But the source to train your own LLM equivalent is also released though (minus the data). Hence why there are so many variants of LLaMa. You also can’t fine tune it without the original model structure. The weights give the community a starting point so they don’t need literally millions of dollar worth of compute power to get to the same step.
  - monocasa 3y ago
    Would Mistral's engineers be satisfied with the release if they had to rebuild from scratch?
    - syntaxing 3y ago
      But they built a llama equivalent + some enhancements that gives better performance…I’m not sure if this would be possible at all without Meta releasing all the required code and paper for LLaMa to begin with.
      - monocasa 3y ago
        Meta didn't release all of the required code to build LLaMa, just enough run inference with their weights.
    - godelski 3y ago
      > Would Mistral's engineers be satisfied with the release if they had to rebuild from scratch? Yeah, probably. But depends on what you're asking. In the exact same method to get the exact same results down to epsilon error? (Again, ML models are not deterministic) Probably not. This honestly can even change with a different version of pytorch, but yes, knowing the HPs would help get closer. But to train another 7B model of the __exact__ same architecture? Yeah, definitely they've provided all you need for that. You can take this model and train it from scratch on any data you want and train it in any way you want.
      - monocasa 3y ago
        I didn't ask if they'd be able to make do. I asked if they'd be satisfied. Also, wrt > Again, ML models are not deterministic) ML models are absolutely deterministic if you have the discipline to do so (which is necessary at higher scale ML work when hardware is stochastically flaky).
- jncfhnb 3y ago
  It’s not a build artifact. The algorithm is the open source code.
  - monocasa 3y ago
    So the engineers that built this edited this release directly? Or did they write code that built this?
    - jncfhnb 3y ago
      Every engineer uses tools to create their products. Those products that are open source do not include the tools used to create them (usually).
      - monocasa 3y ago
        "Open source" normally contains what the engineers working on it edit to submit a job to build into the build artifacts. This does not contain that, but instead the result of the build process.
        jncfhnb 3y ago
        That’s because you’re failing to differentiate a build process meaning compiling your stuff to an end result, vs. fine tuning an algorithm to refine code.
        monocasa 3y ago
        Explicitly. I regard the distinction as meaningless and haven't heard a good reason as why I should reconsider when the process behind creating the weights is so integral to the overall engineering process here.
        jncfhnb 3y ago
        Well fortunately that’s just your opinion and it is neither popular nor relevant. To quote stallman > The "source code" for a work means the preferred form of the work for making modifications to it That’s what this is. This is not a work that produces weights. So the code related to making weights would not be the source code of this project. It would be the source code of the tools used to make this project.

monocasa 3y ago

> It's open source. How did the engineers who built it do so? Is there more source to create this build artifact?

syntaxing 3y ago
But the source to train your own LLM equivalent is also released though (minus the data). Hence why there are so many variants of LLaMa. You also can’t fine tune it without the original model structure. The weights give the community a starting point so they don’t need literally millions of dollar worth of compute power to get to the same step.
- monocasa 3y ago
  Would Mistral's engineers be satisfied with the release if they had to rebuild from scratch?
  - syntaxing 3y ago
    But they built a llama equivalent + some enhancements that gives better performance…I’m not sure if this would be possible at all without Meta releasing all the required code and paper for LLaMa to begin with.
    - monocasa 3y ago
      Meta didn't release all of the required code to build LLaMa, just enough run inference with their weights.
  - godelski 3y ago
    > Would Mistral's engineers be satisfied with the release if they had to rebuild from scratch? Yeah, probably. But depends on what you're asking. In the exact same method to get the exact same results down to epsilon error? (Again, ML models are not deterministic) Probably not. This honestly can even change with a different version of pytorch, but yes, knowing the HPs would help get closer. But to train another 7B model of the __exact__ same architecture? Yeah, definitely they've provided all you need for that. You can take this model and train it from scratch on any data you want and train it in any way you want.
    - monocasa 3y ago
      I didn't ask if they'd be able to make do. I asked if they'd be satisfied. Also, wrt > Again, ML models are not deterministic) ML models are absolutely deterministic if you have the discipline to do so (which is necessary at higher scale ML work when hardware is stochastically flaky).
jncfhnb 3y ago
It’s not a build artifact. The algorithm is the open source code.
- monocasa 3y ago
  So the engineers that built this edited this release directly? Or did they write code that built this?
  - jncfhnb 3y ago
    Every engineer uses tools to create their products. Those products that are open source do not include the tools used to create them (usually).
    - monocasa 3y ago
      "Open source" normally contains what the engineers working on it edit to submit a job to build into the build artifacts. This does not contain that, but instead the result of the build process.
      - jncfhnb 3y ago
        That’s because you’re failing to differentiate a build process meaning compiling your stuff to an end result, vs. fine tuning an algorithm to refine code.
        monocasa 3y ago
        Explicitly. I regard the distinction as meaningless and haven't heard a good reason as why I should reconsider when the process behind creating the weights is so integral to the overall engineering process here.
        jncfhnb 3y ago
        Well fortunately that’s just your opinion and it is neither popular nor relevant. To quote stallman > The "source code" for a work means the preferred form of the work for making modifications to it That’s what this is. This is not a work that produces weights. So the code related to making weights would not be the source code of this project. It would be the source code of the tools used to make this project.