3 ms·
Image models support merging already. There exists thousands of StableDiffusion models for that reason only. Downside is that almost all models you see are now
by smusamashah 3y ago
Image models support merging already. There exists thousands of StableDiffusion models for that reason only. Downside is that almost all models you see are now inbred. Community does talk about it and can see the effect this is having on the image quality. A prominent example is you will see the exact same japanese kind of girl face from almost all models out there when you generate image of a woman. Check out Civitai to see what I am talking about. It's not easy to train new models but super easy to merge.
We can expect an explosion of LLMs using this technique similarly. And later at some point the degradation of quality perhaps. Or may be all those LLMs will be just saying the same things at that point.
- famouswaffles 3y agoImage models merging is kind of meh though. You can't really merge two SD models trained on different styles with different keywords and get a model that knows both independently.
- smusamashah 3y agoNever done that myself but always thought that was the point of these merges. May be it doesn't understand keywords after merge but I think they do keep the styles in some form. What's the point of merges if that wasn't possible. People sometimes share how much of a model they merged by a factor.
- famouswaffles 3y agoLet's say i trained a model on the artworks of fiona staples that is invoked by typing "fiona staples style". Then i trained a model on the artworks of james daly that is invoked by "james daly style". What i want when i merge such models is a model that can generate in the artsyle of either fiona and james independently or a mix of both if i specify both keywords in the same prompt. Currently, if you merge these 2 models and generate "busy city street, fiona staples style", you will not get a model that can generate works in fiona's style, you will just get a model that will generate an odd mix of fiona and daly even if you only specify one of them. It means you either need to train a million different models for a million different concepts with no chance of cross usage (e.g x person wearing y clothes in z style will not be possible) or train those concepts at the same time, which becomes very cumbersome requiring a retrain of n+1 concepts on a fresh model anytime you want to introduce a new concept. Oh, and training on x then training on y doesn't work in practice either because the model will mostly forget x learning y.
- zingelshuher 3y agoIntuitively looks like models should be close enough, or sparse enough for merge to work. I wonder if MoE experts can be merged(?)