6 ms·

For reference, here's what you can get with a properly tweaked Stable Diffusion, all running locally on my PC. Can be set up on almost any PC with a mid range G

For reference, here's what you can get with a properly tweaked Stable Diffusion, all running locally on my PC. Can be set up on almost any PC with a mid range GPU in a few minutes if you know what you're doing. I didn't do any cherry picking; this is the first thing it generated. 4 images per prompt. 1st prompt: https://i.postimg.cc/T3nZ9bQy/1st.png https://i.postimg.cc/T3nZ9bQy/1st.png 2nd prompt: https://i.postimg.cc/XNFm3dSs/2nd.png https://i.postimg.cc/XNFm3dSs/2nd.png 3rd prompt: https://i.postimg.cc/c1bCyqWR/3rd.png https://i.postimg.cc/c1bCyqWR/3rd.png

jfdi 3y ago
Nice! Would you mind sharing which stable diff you used / where you obtained from?
- kouteiheika 3y ago
  I'm using my own custom trained model. Here, I've uploaded it to civitai: https://civitai.com/models/94176 https://civitai.com/models/94176 There are plenty of other good models too though.
  - bavell 3y ago
    Any tips or guides you followed on training your custom model? I've done a few LoRAs and TI but haven't gotten to my own models yet. Your results look great and I'd love a little insight into how you arrived there and what methods/tools you used.
    - kouteiheika 3y ago
      I'm not an expert at this and there are probably better ways to do this/might not work for you/your mileage may vary, so please take this with a huge grain of salt, but roughly this worked for me: 1. Start with a good base model(s) from which to train from. 2. Have a lot of diverse images. 3. Ideally train for only one epoch. (Having a lot of images helps here.) 4. If you get bad results lower the learning rate and try again. 5. After training try to mix your finetuned model with the original one, in steps of 10%, generate X/Y plot of it, pick the best result. 6. Repeat this process as long as you're getting an improvement. For training I mostly used scripts from here: https://github.com/bmaltais/kohya_ss https://github.com/bmaltais/kohya_ss The main problem here is that essentially during inference you're using a bag of tricks to make the output better (e.g. good negative embeddings), but when training you don't. (And I'm not entirely sure how you'd actually integrate those into the training process; might be possible, but I didn't want to spend too much time on it.) So your fine tuning as-is might improve the output of the model when no tricks are used, but it can also regress it when the tricks are used. Which I why I did the "mix and pick the best one" step. But, again, I'm not an expert at this and just did this for fun. Ultimately there might be better ways to do it.
      - bavell 3y ago
        Great tips, thank you! It feels like I'm right behind you in terms of where I'm at so your input is very much appreciated. 3. Train for only 1 epoch - interesting, any known rationale here? 5. I just read somewhere else that someone got good results from mixing their custom model with the original (60/40 in their case) - good to hear some more anecdotes that this is pretty effective. Especially the further training after merging, sounds promising! I've also been using kohya_ss for training LoRAs so great to hear it works for you for models as well. On your point about the inference tricks, definitely noted but I did notice that you can feed some params (# of samples, negative embeddings, etc) to the sample images generated during training (check the textarea placeholder text). Still not going to have all usual the tricks but it'll get you a little closer.
    - zirgs 3y ago
      Make sure that you have enough vram. I can train loras with 8 gb easily, but when I tried to train a model - it gives me an oom error.
      - bavell 3y ago
        Hopefully my 12GB is enough!
bluetidepro 3y ago
Do you have any good tutorial links to setup Stable Diffusion locally?
muhammadusman 3y ago
thanks for doing this, I would like to include these into the blog post as well. Can I use these and credit you for them? (let me know what you'd like linked)
- kouteiheika 3y ago
  Sure. No need to credit me.
  - muhammadusman 3y ago
    thanks, updated the post with your results as well :)
ewjt 3y ago
Can you elaborate on “properly tweaked”? When I use one of the Stable Diffusion and AUTOMATIC1111 templates on runpod.io, the results are absolutely worthless. This is using some of the popular prompts you can find on sites like prompthero that show amazing examples. It’s been serious expectation vs. reality disappointment for me and so I just pay the MidJourney or DALL-E fees.
- capybara_2020 3y ago
  First off are you using a custom model or the default SD model? The default model is not the greatest. Have you tried controlnet? But yes SD can be a bit of a pain to use. Think of it like this. SD = Linux, Midjourney = Windows/MacOS. SD is more powerful and user controllable but that also means it has a steeper learning curve.
- orbital-decay 3y ago
  Are you using txt2img with the vanilla model? SD's actual value is in the large array of higher-order input methods and tooling; as a tradeoff, it requires more knowledge. Similarly to 3D CGI, it's a highly technical area. You don't just enter the prompt with it. You can finetune it on your own material, or choose one of the hundreds of public finetuned models. You can guide it in a precise manner with a sketch or by extracting a pose from a photo using controlnets or any other method. You can influence the colors. You can explicitly separate prompt parts so the tokens don't leak into each other. You can use it as a photobashing tool with a plugin to popular image editing software. Things like ComfyUI enable extremely complicated pipelines as well. etc etc etc
  - nomand 3y ago
    Is there a coherent resource (not a scattered 'just google it' series of guides from all over the place) that encapsulates some of the concepts and workflows you're describing? What would be the best learning site/resource for arriving at understanding how to integrate and manipulate SD with precision like that? Thanks
    - 3y ago
troupo 3y ago
"Just" use a "properly tweaked" something.
senko 3y ago
I am sure you're right, but "if you know what you're doing" does a lot of heavy lifting here. We could just as easily say "hosting your own email can be set up in a few minutes if you know what you're doing". I could do that, but I couldn't get local SD to generate comparable images if my life depended on it.
- caseyf 3y ago
  If you have an apple device, there is free GUI for Stable Diffusion called "Draw Things. It is nice and it just works. https://apps.apple.com/us/app/6444050820 https://apps.apple.com/us/app/6444050820 screenshot of the options interface: https://stash.cass.xyz/drawthings-1687292611.png https://stash.cass.xyz/drawthings-1687292611.png
  - ProfessorLayton 3y ago
    Wow. It's both amazing and for some reason horrifying that this can run on an iPhone 11 (non-pro), and at reasonable speeds!
deleted 3y ago
[deleted]
tambourine_man 3y ago
Those are amazing, please consider writing a blog post of the steps you did to install and tweak Stable Diffusion to achieve these results. I'm sure many of us would love to read it.
bibanez 3y ago
You got incorporated into the article! Nice.

jfdi 3y ago

Nice! Would you mind sharing which stable diff you used / where you obtained from?

kouteiheika 3y ago
I'm using my own custom trained model. Here, I've uploaded it to civitai: https://civitai.com/models/94176 https://civitai.com/models/94176 There are plenty of other good models too though.
- bavell 3y ago
  Any tips or guides you followed on training your custom model? I've done a few LoRAs and TI but haven't gotten to my own models yet. Your results look great and I'd love a little insight into how you arrived there and what methods/tools you used.
  - kouteiheika 3y ago
    I'm not an expert at this and there are probably better ways to do this/might not work for you/your mileage may vary, so please take this with a huge grain of salt, but roughly this worked for me: 1. Start with a good base model(s) from which to train from. 2. Have a lot of diverse images. 3. Ideally train for only one epoch. (Having a lot of images helps here.) 4. If you get bad results lower the learning rate and try again. 5. After training try to mix your finetuned model with the original one, in steps of 10%, generate X/Y plot of it, pick the best result. 6. Repeat this process as long as you're getting an improvement. For training I mostly used scripts from here: https://github.com/bmaltais/kohya_ss https://github.com/bmaltais/kohya_ss The main problem here is that essentially during inference you're using a bag of tricks to make the output better (e.g. good negative embeddings), but when training you don't. (And I'm not entirely sure how you'd actually integrate those into the training process; might be possible, but I didn't want to spend too much time on it.) So your fine tuning as-is might improve the output of the model when no tricks are used, but it can also regress it when the tricks are used. Which I why I did the "mix and pick the best one" step. But, again, I'm not an expert at this and just did this for fun. Ultimately there might be better ways to do it.
    - bavell 3y ago
      Great tips, thank you! It feels like I'm right behind you in terms of where I'm at so your input is very much appreciated. 3. Train for only 1 epoch - interesting, any known rationale here? 5. I just read somewhere else that someone got good results from mixing their custom model with the original (60/40 in their case) - good to hear some more anecdotes that this is pretty effective. Especially the further training after merging, sounds promising! I've also been using kohya_ss for training LoRAs so great to hear it works for you for models as well. On your point about the inference tricks, definitely noted but I did notice that you can feed some params (# of samples, negative embeddings, etc) to the sample images generated during training (check the textarea placeholder text). Still not going to have all usual the tricks but it'll get you a little closer.
  - zirgs 3y ago
    Make sure that you have enough vram. I can train loras with 8 gb easily, but when I tried to train a model - it gives me an oom error.
    - bavell 3y ago
      Hopefully my 12GB is enough!

bluetidepro 3y ago

Do you have any good tutorial links to setup Stable Diffusion locally?

muhammadusman 3y ago

thanks for doing this, I would like to include these into the blog post as well. Can I use these and credit you for them? (let me know what you'd like linked)

kouteiheika 3y ago
Sure. No need to credit me.
- muhammadusman 3y ago
  thanks, updated the post with your results as well :)

ewjt 3y ago

Can you elaborate on “properly tweaked”? When I use one of the Stable Diffusion and AUTOMATIC1111 templates on runpod.io, the results are absolutely worthless. This is using some of the popular prompts you can find on sites like prompthero that show amazing examples. It’s been serious expectation vs. reality disappointment for me and so I just pay the MidJourney or DALL-E fees.

capybara_2020 3y ago
First off are you using a custom model or the default SD model? The default model is not the greatest. Have you tried controlnet? But yes SD can be a bit of a pain to use. Think of it like this. SD = Linux, Midjourney = Windows/MacOS. SD is more powerful and user controllable but that also means it has a steeper learning curve.
orbital-decay 3y ago
Are you using txt2img with the vanilla model? SD's actual value is in the large array of higher-order input methods and tooling; as a tradeoff, it requires more knowledge. Similarly to 3D CGI, it's a highly technical area. You don't just enter the prompt with it. You can finetune it on your own material, or choose one of the hundreds of public finetuned models. You can guide it in a precise manner with a sketch or by extracting a pose from a photo using controlnets or any other method. You can influence the colors. You can explicitly separate prompt parts so the tokens don't leak into each other. You can use it as a photobashing tool with a plugin to popular image editing software. Things like ComfyUI enable extremely complicated pipelines as well. etc etc etc
- nomand 3y ago
  Is there a coherent resource (not a scattered 'just google it' series of guides from all over the place) that encapsulates some of the concepts and workflows you're describing? What would be the best learning site/resource for arriving at understanding how to integrate and manipulate SD with precision like that? Thanks
  - 3y ago

troupo 3y ago

"Just" use a "properly tweaked" something.

senko 3y ago

I am sure you're right, but "if you know what you're doing" does a lot of heavy lifting here. We could just as easily say "hosting your own email can be set up in a few minutes if you know what you're doing". I could do that, but I couldn't get local SD to generate comparable images if my life depended on it.

caseyf 3y ago
If you have an apple device, there is free GUI for Stable Diffusion called "Draw Things. It is nice and it just works. https://apps.apple.com/us/app/6444050820 https://apps.apple.com/us/app/6444050820 screenshot of the options interface: https://stash.cass.xyz/drawthings-1687292611.png https://stash.cass.xyz/drawthings-1687292611.png
- ProfessorLayton 3y ago
  Wow. It's both amazing and for some reason horrifying that this can run on an iPhone 11 (non-pro), and at reasonable speeds!

deleted 3y ago

[deleted]

tambourine_man 3y ago

Those are amazing, please consider writing a blog post of the steps you did to install and tweak Stable Diffusion to achieve these results. I'm sure many of us would love to read it.

bibanez 3y ago

You got incorporated into the article! Nice.