I tried to touch on that in my comment. I've used training methods such as LoRA: Low-Rank Adaptation of Large Language Models. I feel they work well to train it

I tried to touch on that in my comment. I've used training methods such as LoRA: Low-Rank Adaptation of Large Language Models. I feel they work well to train it on a specific subject, but it's at the cost of over-riding other weights in the model, and it tends to distort things that are unrelated because what a ML model considered to be related is likely not at all what we as humans believe to be related. I've tried LoRA in the context of stable diffusion only. You can see images using LoRAs can be tricky, and even harder to combine 2 LoRAs etc. Adding too many LoRA modifiers can straight up give distorted messed up images.