4 ms·

It's a little bit more work to set up, but I'd recommend Krita with diffusion if you want to get into this. Runs locally, swap exact checkpoints you want to use

It's a little bit more work to set up, but I'd recommend Krita with diffusion if you want to get into this. Runs locally, swap exact checkpoints you want to use, supports poseable controlnets, and it doesn't phone home to mommy Microsoft. https://github.com/Acly/krita-ai-diffusion https://github.com/Acly/krita-ai-diffusion Video demonstration https://youtu.be/AF2VyqSApjA https://youtu.be/AF2VyqSApjA

hot_gril 2y ago
I was about to ask how you do the equivalent in Krita, but I saw the GitHub showed that under "Using ControlNet to guide image generation with a crude scribble". Cool, gonna try this on my Mac mini.
3abiton 2y ago
I got myself recently one of those cheap graphic tablets (Huion), for some drawing and note taking, and discovered Krita. Absolutely amazing open-source software. And integration with SD is just amazing.
- noman-land 2y ago
  Agreed. I was looking for a FLOSS Photoshop and stumbled upon this gem. Outstanding piece of work, this is. Up there with the greats like Blender, DaVinci Resolve, VLC.
  - squigz 2y ago
    Sharing this for absolutely no reason ;) https://krita.org/en/donations/ https://krita.org/en/donations/
skybrian 2y ago
Maybe someone could explain what I'm doing wrong? Using the online service, I couldn't figure out how to get it to generate an image from a doodle using img2img. Generating an image from a text prompt works, but that's nothing new. Running locally, it was way too slow on my M2 Mac Air.
wruza 2y ago
It was still too cumbersome to set up and use last time I checked. And most of these integrations are just model+prompt without any common features that sdwebui (or comfyui) has. It looks cool as a proof of concept, but isn’t good, cause it’s a poor model+controlnet+segmentation ui. All they have to do is to add a few fields and checkboxes, but don’t expect it soon. I don’t even think that an image editor is a correct starting point for this coop. Imo, it’s webui.
aimor 2y ago
I've had a pretty awesome experience with this, even with an old 8 GB AMD card. It's not perfect, but being able to run models locally within the paint app while mixing different control layers feels like the future.
- Rinzler89 2y ago
  >even with an old 8 GB AMD card That's the main bummer here, not the age of the GPU, but the need for a dedicated one. When most people nowadays have laptops, and only with integrated GPUs to boot, that's a large demographic being excluded, even with new machines. Maybe with the new gen of chips with NPUs from Qualcomm, Intel and AMD this will slowly change but they just dropped and I personally won't exchange my 9 month old laptop for a new one just for that feature alone, and I suspect most PC users will behave the same considering the multi year long upgrade cycles in this market. So it will probably take a long time till on-device generative AI becomes mainstream on the PC.
  - numpad0 2y ago
    That's where Qualcomm/Intel/Microsoft "AI PC" branding with 40 TOPS bar comes in. Besides, GPU had been a requirement for AI for years by now.
  - mft_ 2y ago
    For decent on-device inference, you need enough memory of high-enough bandwidth and enough processing power connected to that memory. The traditional PC architecture is ill-suited to this, which is why for most computers, a GPU (which offers all three in a single package) is currently the best approach... only at the moment, even a 4090 only offers enough memory for moderately-sized models to load and run. The architecture that supports Apple's ARM computers is (by design or happenstance) far better suited: unified memory maximises the model that can be loaded, some higher-end options have decently-high memory bandwidth, and the architecture lets any of the different processing units access the unified memory. Their weakness is cost, and that the processors aren't powerful enough yet to compete at the top end. So there's currently an empty middle-ground to be won, and it's interesting to watch and wait to see how it will be won. e.g. - affordable GPUs with much larger memory for the big models? (i.e. GPUs, but optimised) - affordable unified memory computers with more processing power (i.e. Apple's approach, but optimised) - something else (probably on the software side) making larger model inference more efficient, either from a processing side (i.e. faster for the same output) or from a memory utilisation side (e.g. loading or streaming parts of large models to cope with smaller device memory, or smaller models for the same output)
noman-land 2y ago
Holy crap it is crazy that this is possible at all, let alone offline and local, let alone free and open source. Just, wow.
HyprMusic 2y ago
Thanks for sharing, I've been struggling to learn Draw Things and this feels much more intuitive.

hot_gril 2y ago

I was about to ask how you do the equivalent in Krita, but I saw the GitHub showed that under "Using ControlNet to guide image generation with a crude scribble". Cool, gonna try this on my Mac mini.

3abiton 2y ago

I got myself recently one of those cheap graphic tablets (Huion), for some drawing and note taking, and discovered Krita. Absolutely amazing open-source software. And integration with SD is just amazing.

noman-land 2y ago
Agreed. I was looking for a FLOSS Photoshop and stumbled upon this gem. Outstanding piece of work, this is. Up there with the greats like Blender, DaVinci Resolve, VLC.
- squigz 2y ago
  Sharing this for absolutely no reason ;) https://krita.org/en/donations/ https://krita.org/en/donations/

skybrian 2y ago

Maybe someone could explain what I'm doing wrong? Using the online service, I couldn't figure out how to get it to generate an image from a doodle using img2img. Generating an image from a text prompt works, but that's nothing new. Running locally, it was way too slow on my M2 Mac Air.

wruza 2y ago

It was still too cumbersome to set up and use last time I checked. And most of these integrations are just model+prompt without any common features that sdwebui (or comfyui) has. It looks cool as a proof of concept, but isn’t good, cause it’s a poor model+controlnet+segmentation ui. All they have to do is to add a few fields and checkboxes, but don’t expect it soon. I don’t even think that an image editor is a correct starting point for this coop. Imo, it’s webui.

aimor 2y ago

I've had a pretty awesome experience with this, even with an old 8 GB AMD card. It's not perfect, but being able to run models locally within the paint app while mixing different control layers feels like the future.

Rinzler89 2y ago
>even with an old 8 GB AMD card That's the main bummer here, not the age of the GPU, but the need for a dedicated one. When most people nowadays have laptops, and only with integrated GPUs to boot, that's a large demographic being excluded, even with new machines. Maybe with the new gen of chips with NPUs from Qualcomm, Intel and AMD this will slowly change but they just dropped and I personally won't exchange my 9 month old laptop for a new one just for that feature alone, and I suspect most PC users will behave the same considering the multi year long upgrade cycles in this market. So it will probably take a long time till on-device generative AI becomes mainstream on the PC.
- numpad0 2y ago
  That's where Qualcomm/Intel/Microsoft "AI PC" branding with 40 TOPS bar comes in. Besides, GPU had been a requirement for AI for years by now.
- mft_ 2y ago
  For decent on-device inference, you need enough memory of high-enough bandwidth and enough processing power connected to that memory. The traditional PC architecture is ill-suited to this, which is why for most computers, a GPU (which offers all three in a single package) is currently the best approach... only at the moment, even a 4090 only offers enough memory for moderately-sized models to load and run. The architecture that supports Apple's ARM computers is (by design or happenstance) far better suited: unified memory maximises the model that can be loaded, some higher-end options have decently-high memory bandwidth, and the architecture lets any of the different processing units access the unified memory. Their weakness is cost, and that the processors aren't powerful enough yet to compete at the top end. So there's currently an empty middle-ground to be won, and it's interesting to watch and wait to see how it will be won. e.g. - affordable GPUs with much larger memory for the big models? (i.e. GPUs, but optimised) - affordable unified memory computers with more processing power (i.e. Apple's approach, but optimised) - something else (probably on the software side) making larger model inference more efficient, either from a processing side (i.e. faster for the same output) or from a memory utilisation side (e.g. loading or streaming parts of large models to cope with smaller device memory, or smaller models for the same output)

noman-land 2y ago

Holy crap it is crazy that this is possible at all, let alone offline and local, let alone free and open source. Just, wow.

HyprMusic 2y ago

Thanks for sharing, I've been struggling to learn Draw Things and this feels much more intuitive.