5 ms·
You can do your own fine tuning on existing models > How do I integrate changes to this model that others have made Typically with a LoRA
by squeaky-clean 3y ago
You can do your own fine tuning on existing models
> How do I integrate changes to this model that others have made
Typically with a LoRA
- monocasa 3y agoIs that how the engineers at Mistral iterated on this model? I can patch windows and other binaries too (I'm actually pretty decent at that; 20 or so years with IDA/ghidra). That doesn't make them open source.
- spullara 3y agoYou don't iterate on the model weights, you train them on billions of tokens. There was no real iteration, you set it up and wait for the GPUs to do the work. The model architecture itself is done before you start training. If you change it, you generally have to start training from scratch. You do get the ability to further train the model using fine-tuning or LoRas.
- monocasa 3y agoThere's absolutely state space iteration in model training. Layer sizes, composition, construction. > There was no real iteration, you set it up and wait for the GPUs to do the work. The model architecture itself is done before you start training. If you change it, you generally have to start training from scratch. That's like saying there's no design iteration in software because you type 'make' and the executable is built.
- spullara 3y agoThe point is that iteration is before training. You don't iterate post starting the training and presumably this person doesn't want to train the model themselves. If they do, they can do that but they need to get the training data and a huge amount of GPUs.
- monocasa 3y agoOnce again, yes, the training cycle is more like a compile step. Design iteration of the shape of the model absolutely occurs with partially trained weights however.
- brucethemoose2 3y agoThey trained it from scratch, but there is nothing stopping you from doing some more training on top of it yourself. Nothing is obfuscated, no more information is required beyond the prompting syntax... they just released basically no documentation, which is unfortunately kinda standard in AI land. There is already a sizable ecosystem of people doing just that: https://huggingface.co/models?sort=modified&search=7B https://huggingface.co/models?sort=modified&search=7B
- monocasa 3y agoJust like Microsoft isn't stopping me from patching Windows. > Nothing is obfuscated The dataset and pipeline to rebuild these weights are not included.
- brucethemoose2 3y ago> The dataset and pipeline to rebuild these weights are not included. This is a good point. But its probably not deterministic and reproducible. I bet the researchers themselves couldn't remake the exact weights (or even similar weights) from scratch.
- monocasa 3y agoI mean most binaries aren't reproducible. That takes a lot of work and isn't worth it most of the time. However, I've heard from ML engineers at the big shops that hermetic, reproducible builds are key to making any progress at scale with ML. Apparently that goes all the way back to when Jeff Dean took over Google Brain.
- Scene_Cast2 3y agoYep. Even if the initial seed for parameter init, the example shuffling seed, etc were constant, the distributed nature of training (and syncing the model across machines) would kill reproducibility. Not to mention resuming from checkpoints after gradient explosions, etc.
- 3y ago