3 ms·
This is all super cool. Some random suggestions: - I dunno what diffusion framework you are using, but the AITemplate (for GPUS) or diffusers JAX (for TPUs) b
by brucethemoose2 3y ago
This is all super cool.
Some random suggestions:
- I dunno what diffusion framework you are using, but the AITemplate (for GPUS) or diffusers JAX (for TPUs) backend can massively increase your diffusion throughput.
- Alternatively, I believe HuggingFace already has a JAX backend for Stable Diffusion XL, so you could run a model with much better support for large resolutions/inpainting massive images at a similar (?) speed.
- There are schemes for area prompting and subject "subset" prompting in stable diffusion, as well as using images as input. As an example of how y'all might use this, you could generate a image for Character A, an image for Character B, encode them. specify that the character A prompt latents go on the left side of the image, and the character B prompt latents go on the right side of the image. And of course you can add to these area prompts, like "jumping" on the left side and "ducking" on the right side of the image. There's also a way to specify which prompts/encoded images belong to which subjects instead of manually cutting out areas, see: https://github.com/BlenderNeko/ComfyUI_Cutoff https://github.com/BlenderNeko/ComfyUI_Cutoff
- Fengjiao 3y agoWoaaa love these inputs. Thank you! Wasn't aware of the JAX backend will check it out. Right now we're on SD 1.5. We tried SDXL but found the quality improvement to be marginal. Yes to area prompting/regional control to help people create more complex scenes. I need some design thinking first since it's easy to over build and spit out something super complicated. Immediate next step is to def add controlnet.
- brucethemoose2 3y ago> JAX Yeah, check out their post: https://huggingface.co/blog/sdxl_jax https://huggingface.co/blog/sdxl_jax I dunno how expensive TPU instances are these days, but the performance is insane! > We tried SDXL but found the quality improvement to be marginal. Yeah, the vanilla HF diffusers pipe is unimpressive to me. Try playing with this though, turn on FreeU and specify an anime style: https://github.com/MoonRide303/Fooocus-MRE https://github.com/MoonRide303/Fooocus-MRE I have never gotten such high quality results from simple prompts, even in cloud models like Midjourney/GPT4. The question is how to port even part of that magic over to the diffusers pipeline...
- brucethemoose2 3y agoAlso, VoltaML has a good reference GPU AITemplate SD 1.5 implementation: https://github.com/VoltaML/voltaML-fast-stable-diffusion/tree/experimental https://github.com/VoltaML/voltaML-fast-stable-diffusion/tre... The speed jump is massive on my desktop GPU, probably even more dramatic on cloud hardware, and it may support some things (weight swapping/lora swapping/resolution changing/controlnet) better than JAX.
- Fengjiao 3y agoMy issue previously with these prebuilt backends is that you can't tweak it like sdwebui does, but to make our thing work it took a thousand tweaks. Can look into this first to see how customizable it is.
- brucethemoose2 3y agoVoltaML is a relatively vanilla diffusers-based backend, so its not a hairy monster to hack like you may have seen with SAI-based UIs (like Comfy, Fooocus and Automatic) The AITTemplate code is a lightly modified version of Facebook's example dynamic AIT script, to get rid of small issues like VRAM spikes: https://github.com/facebookincubator/AITemplate/tree/main/examples/05_stable_diffusion https://github.com/facebookincubator/AITemplate/tree/main/ex... InvokeAI is also diffusers based, but they seem to mess with the pipeline a bit more. Anyway, all that may be better as a reference for interesting features rather than a backend to try and adopt.