3 ms·
Can someone please explain or link to some information about how models are merged? Is this genuinely merging weights mathematically or some kind of distillatio
by jordz 4mo ago
Can someone please explain or link to some information about how models are merged? Is this genuinely merging weights mathematically or some kind of distillation (presumably not if they’ve done zero training as the post suggests).
- calebkaiser 4mo agoThis is a good starting point: https://huggingface.co/docs/peft/developer_guides/model_merging https://huggingface.co/docs/peft/developer_guides/model_merg... But yes, in general, merging refers to techniques that directly blend the weights of different models mathematically. It had a big moment of popularity ~2 years ago, with many so-called "Frankenmodels" popping up on leaderboards. I tend to think of merging as belonging to the same general umbrella as things like "abliteration", or other techniques that surgically modify the weights of a model without a traditional training/tuning loop. Maxime Labonne is a great person to follow if you're interested in this general area.
- jxmorris12 4mo agoThere’s nothing to read. Model A: A_1, …, A_n Model B: B_1, …, B_n C_i = A_i * p + B_i * (1 - p) In other words, it’s just a linear combination of the other models’ weights, per position.
- joe_the_user 4mo agoIt's been a while since I looked at neural networks in detail. Do all the large models have a close enough architecture that this makes sense? Do they have the same number of layers and width? I had thought that each model it's own "secret sauce" of normal and special layers (convolution, max-pooling, something-something) stacked together. Genuinely curious.