2 ms·

I don't know anyone has properly analyzed this, nor how robust such methods are if one is trying to cover it up. Also, I doubt anyone has analyzed the scenario

by gwern 3y ago

I don't know anyone has properly analyzed this, nor how robust such methods are if one is trying to cover it up. Also, I doubt anyone has analyzed the scenario where the warm-started model is then extensively trained for trillions of token (possibly with a cyclical LR) particularly in Chinese - the latent spaces are not perfectly aligned Chinese/English and I'd expect that to change it a lot. (The point of this would be that 'cheating' by warm-starting it should let you avoid a lot of training instabilities and issues early in training, and may get you better quality at the end.)

renonce 3y ago
Sounds like an interesting direction of research. There is a tool called mergekit and the models produced by merging different models is called frankenmerge. Search for “frankenmerge” and you can find a lot of interesting results and models and discussions on what works and what doesn’t. Might be a good idea to check previous experiments with these keywords.