9 ms·

How to Build Your Own AI-Generated Images with ControlNet and Stable Diffusion

191 points by gkeechin 3y ago

drschwabe 3y ago
ControlNet model specifically the scribble ControlNet (and ComfyUI) was major gamechanger for me. Was getting good results with just SD and occassional masking but it would take hours and hours to hone in and composite a complex scene with specific requirements & shapes (with most of the work spent currating the best outputs and then blending them into a scene with Gimp/Inkscape). Masking is unintuitive compared to the scribble which gets similar effect; no need to paint masks (which is disruptive to the natural process of 'drawing' IMO) instead just make a general black and white outline of your scene. Simply dial up/down the conditioning strength to have it more tightly or fuzzily follow that outline. You can also use Gimp's Threshold or Inkscape Trace Bitmap tool to get a decent black & white outline from an existing bitmap to expedite the scribble procedure.
- gkeechin 3y ago
  That's for sure - I think we have seen other kind of edge detector or filter work better for differing use cases, especially around foreground images where you want to retain more information (i.e. images with small nitty-gritty details) In this post, we just seek to showcase the fastest way to do it - and how augmentation may potentially help vary the position!
- fsloth 3y ago
  Comfy ui is really nice. The fact that the node graph is saved as png metadata actually makes node based workflows super fluent and reproducible since all you need to do to get the graph for your image is to drag and drop the result png to the gui. This feels like a huge quality-of-life improvement compared to any other lightweight node tools I’ve used.
  - drschwabe 3y ago
    Yeah the PNG embedded 'drag and drop to restore your config' is brilliant. Reminds me of Fireworks which Adobe killed off (after putting out a decent update or two to be fair) which used PNGs for layers and meta ala PSD format. But its more analogous to a 3D modeller suite like Blender or Maya but with theoretical feature such that you could take a rendered output image and dragndrop it back into the 3D viewport and have it restore all the exact render settings you used instantly back. That would be handy!
- moelf 3y ago
  any tutorial you would recommend? I found https://comfyanonymous.github.io/ComfyUI_examples/controlnet/ https://comfyanonymous.github.io/ComfyUI_examples/controlnet...
  - drschwabe 3y ago
    Yeah that tutorial is decent its what I used to get going. Note that all of the images in those comfy tutorials (except for images of the UI itself) can be dragndropped into ComfyUI and you'll get the entire node layout you can use to understand how it works. Another good resource is civit.ai and specifically look for images that have a comfy UI embedded metadata. I made a feature request that they create a tag for uploaders to flag comfyUI pngs but not sure if they've added that yet. Or caroose Reddit or Discord for people sharing PNGs with comfy embeds. Trying out different models (also avail from civit) is a good way to get an understanding of how swapping out models affects performance and the results. I've been abusing Absolutereality (v1.81) + More Details LORA because its just so damn fast and the results are great for almost any requirement I throw at it. AI moves so fast but I don't even bother updating the models anymore there is just so much potential in the models we already have; more pay off would be mastering other techniques like the depth map Control Net. I would say that above all extensive familiarity with an image editor like Photoshop, Gimp, or Krita - will get you the most mileage particularly if you have specific needs beyond just fun and concepting. AI art makes artists better, people who struggle with image editing will struggle to maximize this new tech just as people who struggle with code will have issues maintaining the code Copilot or ChatGPT is spitting out (versus a coder who will refactor and fine tune before integrating to the rest of their application).
- l33tman 3y ago
  You don't need to go through Gimp or Inkscape, this is built-in to the auto1111 ControlNet UI. You just dump the existing photo there and you can select a bunch of pre-processors like edge-detection or 3D depth extraction, which is then fed into ControlNet to generate a new image. This is super powerful for example visualizing the renovation of an apartment room or house exterior.
  - drschwabe 3y ago
    Will have to play with those more thx for the headsup; I do find however for scribble outlines I like to often draw my own lines by hand instead of an auto-generated one to emphasize the absolute key areas that would not otherwise be auto-identified. Logo and 2D design for example where you may have very specifc text shape that needs be preserved regardless of contrast or perceivable depth.
- shostack 3y ago
  Is there any solution for consistency yet that goes beyond form and structure and gets things like outfits, color, and facial features consistent in an easy way to compose scenes with multiple consistent characters?
  - dragonwriter 3y ago
    LoRA for specific items/characters + regional prompting covers a lot of that area.
Magi604 3y ago
SD outputs have an "uncanny valley" type of quality to them. You just KNOW when an image is from SD. And I have looked at getting started with SD, but the requirements and setup and +-prompting "language" just kind of turned me off the whole thing. Whereas with DALL E you can get some hyper-realistic images from it with very little effort using plain human language. I guess my point is to ask whether SD is worth bothering with at this time when DALL E and Imagen and possibly others are just on the brink of becoming mainstream and just going to get better and better. Clunking together something with SD seems unnecessary when you can generate more results, better results, in a faster way, with less requirements, and without the steep learning curve, by using other methods.
- NBJack 3y ago
  Try SDXL. Find a good negative prompt, then just put a short sentence (starting with the kind of image, such as photograph, render, etc.) describing what you want in the positive prompt. It is much simpler and has fantastic results. Tweak to your hearts desire from there. If you see a part of the scene that looks weird (and you know what it should be) add it to your prompt. For example, if you want "photo of a jungle in South America", and the foliage looks weird, add something like "with lush trees and ferns".
- jyap 3y ago
  One major benefit and the reason why I use the StableDiffusion tools and models is because I can run them at home on my relatively old NVIDIA 2080 GPU with 8GB of VRAM. Costs me nothing (besides electricity). Depends if you value this kind of freedom in life. You can do some things such as colorizing black and white images with the Recolor model. https://huggingface.co/stabilityai/control-lora https://huggingface.co/stabilityai/control-lora
  - NBJack 3y ago
    I have to agree at how convenient and (long term) inexpensive this can be. I may not always get the greatest results right away, but it is fun to come up with some ideas, put them into a prompt iterator (or matrix), and run it overnight. I can tweak it to my heart's content.
tayo42 3y ago
With the luggage example it seems to only generate backgrounds where the lighting makes sense? That's kind of interesting. I was wondering how it would handle the highlight on the right.
- minimaxir 3y ago
  Giving Stable Diffusion constraints forces it to get creative. It’s the best argument against “AI generated images are just collages”.
  - Der_Einzige 3y ago
    This is a general result. For example, ChatGPT struggles hard with following lexical, syntactic, or phonetic constraints in prompts due to the tokenization scheme - see https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai-1 https://paperswithcode.com/paper/most-language-models-can-be... LLMs + Diffusors are super charged when using techniques like constraints, controlnet, regional prompting, and related techniques.
- zamalek 3y ago
  In ComfyUI you could run the image through a style-to-style (sdxl refinement might even pull it off) model to change the lighting without changing the content. Or use another ControlNet. Your workflow can get arbitrarily complex.
rvion 3y ago
I'm building CushyStudio https://github.com/rvion/cushystudio#readme https://github.com/rvion/cushystudio#readme to make Stable Diffusion practical and fun to play with. It's still a bit rough around the corners, and I haven't properly launched it yet, but if you want to play with ControlNets, pre-processors, IP adapters, and all those various SD technologies, it's a pretty fun tool ! I personally use for real-time scribble to image, things like this :) (will post that properly on HN in a few days / week I think, when early feedback will have been properly addressed)
- bavell 3y ago
  Looking forward to your launch, I found cushystudio awhile back (maybe from HN?) and cannibalized some of the type generation code to make my own API wrapper for personal uses. Thanks! I barely got it working in that early alpha but it was super helpful for me as a reference. I'll give it another go now that it's further along, it seemed very promising and I liked your workflow approach
jaggs 3y ago
Artroom.ai is a great option. Free image generation feature, plus a ton of editing features like layers, zoom out etc.
ComputerGuru 3y ago
The word ControlNet doesn’t appear even once in the article?
- DarthNebo 3y ago
  They did use the Canny ControlNet Pipeline
  - ComputerGuru 3y ago
    I think the article was updated with that in response to my comment: > P.S. As pointed out by a fellow HackerNews reader, we clearly forgot to include our code snippet for ControlNet in the article. No other code snippet besides the one added in response uses Canny, at least so far as I can see.
    - DarthNebo 3y ago
      Oh I see, my bad
imranhou 3y ago
The versatility of Stable Diffusion, especially when combined with tools like ControlNet, highlights the advantages of a more controlled image generation process. While DALL-E and others provide ease and speed, the depth of customization and local processing capabilities of SD models cater to those seeking deeper creative control and independence.
- ChatGTP 3y ago
  It is interesting isn't it? Because we have "AI" generating the image, but we still seem to want to "paint" or have control over the creative process. Prompts seem to be a new type of camera, lens or paintbrush.
  - barrkel 3y ago
    There's at least three "levels" you can consider with image generation: composition, facial likeness and style. Prompts are pretty weak at composition and are the strongest point of controlnets - they do a great deal to make up for the weakness. But there are some compositions SD can't find even when given detailed controlnets. Style generality is frequently lost in fine-tuned models. The original dreambooth tried to get around this by generating lots of images of the class to retain generality, but it's time intensive to generate all the extra images (and ideally do some QC on them) and train on them too, so it's not often done.
telegpt 3y ago
[dead]
Alifatisk 3y ago
If I have a large dataset or photos with my face? Can I generate my own images in different places and environments using this?
- gbrits 3y ago
  Yep. Lora's are the easiest way to go. Loads of tutorials on Youtube. This is a good one: https://www.youtube.com/watch?v=70H03cv57-o https://www.youtube.com/watch?v=70H03cv57-o
  - Alifatisk 3y ago
    Looks like it requires good hardware to run this? My GPU is too old for this.
    - loudmax 3y ago
      You typically want an Nvidia GPU with at least 8GB of VRAM to get started with this stuff. You can get away with less, but it will be slow going. More VRAM is better. If you're serious about learning how to use these tools, it's far more affordable to rent a GPU in the cloud. Google Colab even has some free tiers with limited access to GPUs that are significantly more powerful than what you would normally put in a desktop.
      - Alifatisk 3y ago
        Thanks for the advice, I’ll look into the option of renting a gpu sadly. I just want to create my own images like Dall-e 2 but using my face instead.
ChildOfChaos 3y ago
Love all this AI stuff, would love to play more with it, but sadly I'm on a 2015 iMac, great for everything else I do but can't do this stuff. It's pricey to get a windows machine + GPU and the cloud options seem a bit more limited and add up quickly too, but it is amazing tech.
- newswasboring 3y ago
  I have done a bunch of stable diffusion stuff on colab. The free version works if you are lucky enough to get a GPU. Used to happen more often before. But the premium colab isn't badly priced either. Here is a colab link to open comfyUI https://github.com/FurkanGozukara/Stable-Diffusion/blob/main/ColabNotebooks/comfyui_SDXL_colab.ipynb https://github.com/FurkanGozukara/Stable-Diffusion/blob/main...
  - ChildOfChaos 3y ago
    They blocked this now on the free version of colab sadly.
imranq 3y ago
While SD is pretty interesting, I'm curious what do people use it for? Outside of custom greeting cards and backgrounds, it's not really precise enough for conceptual art nor is it consistent enough for animation.
- Filligree 3y ago
  Illustrating my fanfiction.

191 points by gkeechin 3y ago

ControlNet model specifically the scribble ControlNet (and ComfyUI) was major gamechanger for me. Was getting good results with just SD and occassional masking but it would take hours and hours to hone in and composite a complex scene with specific requirements & shapes (with most of the work spent currating the best outputs and then blending them into a scene with Gimp/Inkscape). Masking is unintuitive compared to the scribble which gets similar effect; no need to paint masks (which is disruptive to the natural process of 'drawing' IMO) instead just make a general black and white outline of your scene. Simply dial up/down the conditioning strength to have it more tightly or fuzzily follow that outline. You can also use Gimp's Threshold or Inkscape Trace Bitmap tool to get a decent black & white outline from an existing bitmap to expedite the scribble procedure.

gkeechin 3y ago
That's for sure - I think we have seen other kind of edge detector or filter work better for differing use cases, especially around foreground images where you want to retain more information (i.e. images with small nitty-gritty details) In this post, we just seek to showcase the fastest way to do it - and how augmentation may potentially help vary the position!
fsloth 3y ago
Comfy ui is really nice. The fact that the node graph is saved as png metadata actually makes node based workflows super fluent and reproducible since all you need to do to get the graph for your image is to drag and drop the result png to the gui. This feels like a huge quality-of-life improvement compared to any other lightweight node tools I’ve used.
- drschwabe 3y ago
  Yeah the PNG embedded 'drag and drop to restore your config' is brilliant. Reminds me of Fireworks which Adobe killed off (after putting out a decent update or two to be fair) which used PNGs for layers and meta ala PSD format. But its more analogous to a 3D modeller suite like Blender or Maya but with theoretical feature such that you could take a rendered output image and dragndrop it back into the 3D viewport and have it restore all the exact render settings you used instantly back. That would be handy!
moelf 3y ago
any tutorial you would recommend? I found https://comfyanonymous.github.io/ComfyUI_examples/controlnet/ https://comfyanonymous.github.io/ComfyUI_examples/controlnet...
- drschwabe 3y ago
  Yeah that tutorial is decent its what I used to get going. Note that all of the images in those comfy tutorials (except for images of the UI itself) can be dragndropped into ComfyUI and you'll get the entire node layout you can use to understand how it works. Another good resource is civit.ai and specifically look for images that have a comfy UI embedded metadata. I made a feature request that they create a tag for uploaders to flag comfyUI pngs but not sure if they've added that yet. Or caroose Reddit or Discord for people sharing PNGs with comfy embeds. Trying out different models (also avail from civit) is a good way to get an understanding of how swapping out models affects performance and the results. I've been abusing Absolutereality (v1.81) + More Details LORA because its just so damn fast and the results are great for almost any requirement I throw at it. AI moves so fast but I don't even bother updating the models anymore there is just so much potential in the models we already have; more pay off would be mastering other techniques like the depth map Control Net. I would say that above all extensive familiarity with an image editor like Photoshop, Gimp, or Krita - will get you the most mileage particularly if you have specific needs beyond just fun and concepting. AI art makes artists better, people who struggle with image editing will struggle to maximize this new tech just as people who struggle with code will have issues maintaining the code Copilot or ChatGPT is spitting out (versus a coder who will refactor and fine tune before integrating to the rest of their application).
l33tman 3y ago
You don't need to go through Gimp or Inkscape, this is built-in to the auto1111 ControlNet UI. You just dump the existing photo there and you can select a bunch of pre-processors like edge-detection or 3D depth extraction, which is then fed into ControlNet to generate a new image. This is super powerful for example visualizing the renovation of an apartment room or house exterior.
- drschwabe 3y ago
  Will have to play with those more thx for the headsup; I do find however for scribble outlines I like to often draw my own lines by hand instead of an auto-generated one to emphasize the absolute key areas that would not otherwise be auto-identified. Logo and 2D design for example where you may have very specifc text shape that needs be preserved regardless of contrast or perceivable depth.
shostack 3y ago
Is there any solution for consistency yet that goes beyond form and structure and gets things like outfits, color, and facial features consistent in an easy way to compose scenes with multiple consistent characters?
- dragonwriter 3y ago
  LoRA for specific items/characters + regional prompting covers a lot of that area.

Magi604 3y ago

SD outputs have an "uncanny valley" type of quality to them. You just KNOW when an image is from SD. And I have looked at getting started with SD, but the requirements and setup and +-prompting "language" just kind of turned me off the whole thing. Whereas with DALL E you can get some hyper-realistic images from it with very little effort using plain human language. I guess my point is to ask whether SD is worth bothering with at this time when DALL E and Imagen and possibly others are just on the brink of becoming mainstream and just going to get better and better. Clunking together something with SD seems unnecessary when you can generate more results, better results, in a faster way, with less requirements, and without the steep learning curve, by using other methods.

NBJack 3y ago
Try SDXL. Find a good negative prompt, then just put a short sentence (starting with the kind of image, such as photograph, render, etc.) describing what you want in the positive prompt. It is much simpler and has fantastic results. Tweak to your hearts desire from there. If you see a part of the scene that looks weird (and you know what it should be) add it to your prompt. For example, if you want "photo of a jungle in South America", and the foliage looks weird, add something like "with lush trees and ferns".
jyap 3y ago
One major benefit and the reason why I use the StableDiffusion tools and models is because I can run them at home on my relatively old NVIDIA 2080 GPU with 8GB of VRAM. Costs me nothing (besides electricity). Depends if you value this kind of freedom in life. You can do some things such as colorizing black and white images with the Recolor model. https://huggingface.co/stabilityai/control-lora https://huggingface.co/stabilityai/control-lora
- NBJack 3y ago
  I have to agree at how convenient and (long term) inexpensive this can be. I may not always get the greatest results right away, but it is fun to come up with some ideas, put them into a prompt iterator (or matrix), and run it overnight. I can tweak it to my heart's content.

tayo42 3y ago

With the luggage example it seems to only generate backgrounds where the lighting makes sense? That's kind of interesting. I was wondering how it would handle the highlight on the right.

minimaxir 3y ago
Giving Stable Diffusion constraints forces it to get creative. It’s the best argument against “AI generated images are just collages”.
- Der_Einzige 3y ago
  This is a general result. For example, ChatGPT struggles hard with following lexical, syntactic, or phonetic constraints in prompts due to the tokenization scheme - see https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai-1 https://paperswithcode.com/paper/most-language-models-can-be... LLMs + Diffusors are super charged when using techniques like constraints, controlnet, regional prompting, and related techniques.
zamalek 3y ago
In ComfyUI you could run the image through a style-to-style (sdxl refinement might even pull it off) model to change the lighting without changing the content. Or use another ControlNet. Your workflow can get arbitrarily complex.

rvion 3y ago

I'm building CushyStudio https://github.com/rvion/cushystudio#readme https://github.com/rvion/cushystudio#readme to make Stable Diffusion practical and fun to play with. It's still a bit rough around the corners, and I haven't properly launched it yet, but if you want to play with ControlNets, pre-processors, IP adapters, and all those various SD technologies, it's a pretty fun tool ! I personally use for real-time scribble to image, things like this :) (will post that properly on HN in a few days / week I think, when early feedback will have been properly addressed)

bavell 3y ago
Looking forward to your launch, I found cushystudio awhile back (maybe from HN?) and cannibalized some of the type generation code to make my own API wrapper for personal uses. Thanks! I barely got it working in that early alpha but it was super helpful for me as a reference. I'll give it another go now that it's further along, it seemed very promising and I liked your workflow approach

jaggs 3y ago

Artroom.ai is a great option. Free image generation feature, plus a ton of editing features like layers, zoom out etc.

ComputerGuru 3y ago

The word ControlNet doesn’t appear even once in the article?

DarthNebo 3y ago
They did use the Canny ControlNet Pipeline
- ComputerGuru 3y ago
  I think the article was updated with that in response to my comment: > P.S. As pointed out by a fellow HackerNews reader, we clearly forgot to include our code snippet for ControlNet in the article. No other code snippet besides the one added in response uses Canny, at least so far as I can see.
  - DarthNebo 3y ago
    Oh I see, my bad

imranhou 3y ago

The versatility of Stable Diffusion, especially when combined with tools like ControlNet, highlights the advantages of a more controlled image generation process. While DALL-E and others provide ease and speed, the depth of customization and local processing capabilities of SD models cater to those seeking deeper creative control and independence.

ChatGTP 3y ago
It is interesting isn't it? Because we have "AI" generating the image, but we still seem to want to "paint" or have control over the creative process. Prompts seem to be a new type of camera, lens or paintbrush.
- barrkel 3y ago
  There's at least three "levels" you can consider with image generation: composition, facial likeness and style. Prompts are pretty weak at composition and are the strongest point of controlnets - they do a great deal to make up for the weakness. But there are some compositions SD can't find even when given detailed controlnets. Style generality is frequently lost in fine-tuned models. The original dreambooth tried to get around this by generating lots of images of the class to retain generality, but it's time intensive to generate all the extra images (and ideally do some QC on them) and train on them too, so it's not often done.

telegpt 3y ago

[dead]

Alifatisk 3y ago

If I have a large dataset or photos with my face? Can I generate my own images in different places and environments using this?

gbrits 3y ago
Yep. Lora's are the easiest way to go. Loads of tutorials on Youtube. This is a good one: https://www.youtube.com/watch?v=70H03cv57-o https://www.youtube.com/watch?v=70H03cv57-o
- Alifatisk 3y ago
  Looks like it requires good hardware to run this? My GPU is too old for this.
  - loudmax 3y ago
    You typically want an Nvidia GPU with at least 8GB of VRAM to get started with this stuff. You can get away with less, but it will be slow going. More VRAM is better. If you're serious about learning how to use these tools, it's far more affordable to rent a GPU in the cloud. Google Colab even has some free tiers with limited access to GPUs that are significantly more powerful than what you would normally put in a desktop.
    - Alifatisk 3y ago
      Thanks for the advice, I’ll look into the option of renting a gpu sadly. I just want to create my own images like Dall-e 2 but using my face instead.

ChildOfChaos 3y ago

Love all this AI stuff, would love to play more with it, but sadly I'm on a 2015 iMac, great for everything else I do but can't do this stuff. It's pricey to get a windows machine + GPU and the cloud options seem a bit more limited and add up quickly too, but it is amazing tech.

newswasboring 3y ago
I have done a bunch of stable diffusion stuff on colab. The free version works if you are lucky enough to get a GPU. Used to happen more often before. But the premium colab isn't badly priced either. Here is a colab link to open comfyUI https://github.com/FurkanGozukara/Stable-Diffusion/blob/main/ColabNotebooks/comfyui_SDXL_colab.ipynb https://github.com/FurkanGozukara/Stable-Diffusion/blob/main...
- ChildOfChaos 3y ago
  They blocked this now on the free version of colab sadly.

imranq 3y ago

While SD is pretty interesting, I'm curious what do people use it for? Outside of custom greeting cards and backgrounds, it's not really precise enough for conceptual art nor is it consistent enough for animation.

Filligree 3y ago
Illustrating my fanfiction.