4 ms·

They trained it from scratch, but there is nothing stopping you from doing some more training on top of it yourself. Nothing is obfuscated, no more information

They trained it from scratch, but there is nothing stopping you from doing some more training on top of it yourself. Nothing is obfuscated, no more information is required beyond the prompting syntax... they just released basically no documentation, which is unfortunately kinda standard in AI land. There is already a sizable ecosystem of people doing just that: https://huggingface.co/models?sort=modified&search=7B https://huggingface.co/models?sort=modified&search=7B

monocasa 3y ago
Just like Microsoft isn't stopping me from patching Windows. > Nothing is obfuscated The dataset and pipeline to rebuild these weights are not included.
- brucethemoose2 3y ago
  > The dataset and pipeline to rebuild these weights are not included. This is a good point. But its probably not deterministic and reproducible. I bet the researchers themselves couldn't remake the exact weights (or even similar weights) from scratch.
  - monocasa 3y ago
    I mean most binaries aren't reproducible. That takes a lot of work and isn't worth it most of the time. However, I've heard from ML engineers at the big shops that hermetic, reproducible builds are key to making any progress at scale with ML. Apparently that goes all the way back to when Jeff Dean took over Google Brain.
  - Scene_Cast2 3y ago
    Yep. Even if the initial seed for parameter init, the example shuffling seed, etc were constant, the distributed nature of training (and syncing the model across machines) would kill reproducibility. Not to mention resuming from checkpoints after gradient explosions, etc.
    - monocasa 3y ago
      I've heard from ML engineers at larger shops that reproducibility is key to working at scale. That's how you track down "this training regime went to shit because of something we changed" versus "this training regime went to shit because on of the GPUs training it is starting to fail".
- pk-protect-ai 3y ago
  Have you tried the HF version which is linked on the main page? It is transformer based after all and it is converted to HF format. Just use QLoRA to fine-tune whatever you want on the top of that model. They handed you hundreds of GPU hours, without asking anything in return. You may throw it away and train the model from scratch. Be ready to throw $70k-$150k into air warming.
  - monocasa 3y ago
    > Have you tried the HF version which is linked on the main page? It is transformer based after all and it is converted to HF format. Just use QLoRA to fine-tune whatever you want on the top of that model. Is that how their engineers built this model in the first place? Just because they're giving the build artifact of there hard work away doesn't make it open source. > Be ready to throw $70k-$150k into air warming. Just because builds are expensive, doesn't mean releasing the build artifacts counts as open source.
    - pk-protect-ai 3y ago
      [flagged]
      - monocasa 3y ago
        > I see dozens of your posts here complaining about "not open source." You are either a paid provocateur or a troll. I can't reply in my one tree without being called a troll? Also, accusing someone of being a paid provocateur is against HN guidelines. > What exactly is missing in your opinion in this release that is making this model not open source? The source the engineers at Mistral used to build it.
- HanClinto 3y ago
  Is "Doom" open source? The source code to the engine is available and open, but if you want to do anything useful or recreate the original game, you have to provide your own data. This model is open source, much in a similar way that the Doom engine is open source.
  - monocasa 3y ago
    > Is "Doom" open source? > The source code to the engine is available and open, but if you want to do anything useful or recreate the original game, you have to provide your own data. The Doom executable is open source. The data is not. Explicitly, the data is under a different license than the GPL and makes no claims about being open source. There are total conversion mods that replace the closed source levels with open source ones however. > This model is open source, much in a similar way that the Doom engine is open source. Here's the source the engineers that created doom used to build the doom engine: https://github.com/id-Software/DOOM https://github.com/id-Software/DOOM Where is the source used to build this model?
    - HanClinto 3y ago
      The source used to create the model is here: https://github.com/mistralai/mistral-src/blob/main/mistral/model.py https://github.com/mistralai/mistral-src/blob/main/mistral/m... The source used to train information into the model is elsewhere. You have the engine. You don't have the source of the data.
      - monocasa 3y ago
        That's the inference source as stated in the readme > This repository contains minimal code to run our 7B model
        HanClinto 3y ago
        The code I linked is the model source -- the structural skeleton. You can create a model with this source and put whatever weights you want in it -- just like Doom.
artursapek 3y ago
Can you recommend a good resource(s) to learn how to fine-tune one of these on local hardware?
- brucethemoose2 3y ago
  https://github.com/OpenAccess-AI-Collective/axolotl https://github.com/OpenAccess-AI-Collective/axolotl
  - artursapek 3y ago
    Thanks

monocasa 3y ago

Just like Microsoft isn't stopping me from patching Windows. > Nothing is obfuscated The dataset and pipeline to rebuild these weights are not included.

brucethemoose2 3y ago
> The dataset and pipeline to rebuild these weights are not included. This is a good point. But its probably not deterministic and reproducible. I bet the researchers themselves couldn't remake the exact weights (or even similar weights) from scratch.
- monocasa 3y ago
  I mean most binaries aren't reproducible. That takes a lot of work and isn't worth it most of the time. However, I've heard from ML engineers at the big shops that hermetic, reproducible builds are key to making any progress at scale with ML. Apparently that goes all the way back to when Jeff Dean took over Google Brain.
- Scene_Cast2 3y ago
  Yep. Even if the initial seed for parameter init, the example shuffling seed, etc were constant, the distributed nature of training (and syncing the model across machines) would kill reproducibility. Not to mention resuming from checkpoints after gradient explosions, etc.
  - monocasa 3y ago
    I've heard from ML engineers at larger shops that reproducibility is key to working at scale. That's how you track down "this training regime went to shit because of something we changed" versus "this training regime went to shit because on of the GPUs training it is starting to fail".
pk-protect-ai 3y ago
Have you tried the HF version which is linked on the main page? It is transformer based after all and it is converted to HF format. Just use QLoRA to fine-tune whatever you want on the top of that model. They handed you hundreds of GPU hours, without asking anything in return. You may throw it away and train the model from scratch. Be ready to throw $70k-$150k into air warming.
- monocasa 3y ago
  > Have you tried the HF version which is linked on the main page? It is transformer based after all and it is converted to HF format. Just use QLoRA to fine-tune whatever you want on the top of that model. Is that how their engineers built this model in the first place? Just because they're giving the build artifact of there hard work away doesn't make it open source. > Be ready to throw $70k-$150k into air warming. Just because builds are expensive, doesn't mean releasing the build artifacts counts as open source.
  - pk-protect-ai 3y ago
    [flagged]
    - monocasa 3y ago
      > I see dozens of your posts here complaining about "not open source." You are either a paid provocateur or a troll. I can't reply in my one tree without being called a troll? Also, accusing someone of being a paid provocateur is against HN guidelines. > What exactly is missing in your opinion in this release that is making this model not open source? The source the engineers at Mistral used to build it.
HanClinto 3y ago
Is "Doom" open source? The source code to the engine is available and open, but if you want to do anything useful or recreate the original game, you have to provide your own data. This model is open source, much in a similar way that the Doom engine is open source.
- monocasa 3y ago
  > Is "Doom" open source? > The source code to the engine is available and open, but if you want to do anything useful or recreate the original game, you have to provide your own data. The Doom executable is open source. The data is not. Explicitly, the data is under a different license than the GPL and makes no claims about being open source. There are total conversion mods that replace the closed source levels with open source ones however. > This model is open source, much in a similar way that the Doom engine is open source. Here's the source the engineers that created doom used to build the doom engine: https://github.com/id-Software/DOOM https://github.com/id-Software/DOOM Where is the source used to build this model?
  - HanClinto 3y ago
    The source used to create the model is here: https://github.com/mistralai/mistral-src/blob/main/mistral/model.py https://github.com/mistralai/mistral-src/blob/main/mistral/m... The source used to train information into the model is elsewhere. You have the engine. You don't have the source of the data.
    - monocasa 3y ago
      That's the inference source as stated in the readme > This repository contains minimal code to run our 7B model
      - HanClinto 3y ago
        The code I linked is the model source -- the structural skeleton. You can create a model with this source and put whatever weights you want in it -- just like Doom.

artursapek 3y ago

Can you recommend a good resource(s) to learn how to fine-tune one of these on local hardware?

brucethemoose2 3y ago
https://github.com/OpenAccess-AI-Collective/axolotl https://github.com/OpenAccess-AI-Collective/axolotl
- artursapek 3y ago
  Thanks