3 ms·
> 01.AI claims that, the architecture aside, they didn't simply further train LLaMA models but trained from scratch. You can decide for yourself how much you ar
by visarga 3y ago
> 01.AI claims that, the architecture aside, they didn't simply further train LLaMA models but trained from scratch. You can decide for yourself how much you are willing to believe this.
You can't hide this. The latent space remains mostly fixed after pre-training. It all depends on the seed for the initial random init. Further pre-training won't move it enough. Because of this property, you can even average two fine-tunings from the same parent model, but never on models trained from different seeds.
- sroussey 3y agothe averaging seems like good test for who the parent is.
- gwern 3y agoI don't know anyone has properly analyzed this, nor how robust such methods are if one is trying to cover it up. Also, I doubt anyone has analyzed the scenario where the warm-started model is then extensively trained for trillions of token (possibly with a cyclical LR) particularly in Chinese - the latent spaces are not perfectly aligned Chinese/English and I'd expect that to change it a lot. (The point of this would be that 'cheating' by warm-starting it should let you avoid a lot of training instabilities and issues early in training, and may get you better quality at the end.)
- renonce 3y agoSounds like an interesting direction of research. There is a tool called mergekit and the models produced by merging different models is called frankenmerge. Search for “frankenmerge” and you can find a lot of interesting results and models and discussions on what works and what doesn’t. Might be a good idea to check previous experiments with these keywords.