13 ms·
High-performance image generation using Stable Diffusion in KerasCV
- itronitron 4y agosomeone should compare results with just doing a keyword search on deviantart
- ShamelessC 4y agoNice! I'll take anything over the huggingface version - the API design by huggingface where CLIP is in transformers, everything else is in diffusers...not a great developer experience [unless youre the type of person that likes their python to look like half-baked J2EE).
- rcarmo 4y agoThis is _markedly_ faster than the PyTorch versions I've seen (nothing against the library, just categorizing the implementations). It would be nice to see this including the little quality of life additional models (eye fixes, upscaling, etc.), but I suspect the optimizations are transferrable. Either way, getting 3 images for 25 iterations under 10 seconds (quick Colab test, which is where I've taken to comparing these things) is just ridiculously faster.
- zone411 4y agoWhich GPU did you test on Colab? Are you comparing with one of the fp16 PyTorch versions? Their test shows little improvement on V100. PyTorch is now quite a bit more popular than Keras in research-type code (except when it comes from Google) so I don't know if these enhancements will get ported. This port was done by people working on Keras which is kind of telling - there isn't a lot of outside interest.
- kgwgk 4y agoShould we expect people not working on keras to have the interest and ability to get it to work on keras?
- zone411 4y agoIf these people have existing Keras code they want to integrate or they are interested in developing it further in Keras, then it shouldn't require any insider knowledge to create a Keras version of a small but popular open-source project like this. I am very sure we'd get a PyTorch version made by outsiders quickly if Stable Diffusion was originally released in Keras/TF.
- kgwgk 4y agoWhat is your definition of outsider? We got a Keras version made by Divam Gupta very quickly after Stable Diffusion was released. Is he not an outsider?
- zone411 4y agoFrom what I can tell this Keras version was just released (the date on the post is Sep. 25) and the first author listed is the creator of Keras. Is this incorrect? I am not familiar with Divam Gupta and I would consider outsiders to be people not paid by Google.
- kgwgk 4y agohttps://mobile.twitter.com/divamgupta/status/1571234504320208897 https://mobile.twitter.com/divamgupta/status/157123450432020... https://github.com/divamgupta/stable-diffusion-tensorflow https://github.com/divamgupta/stable-diffusion-tensorflow Now they are working together. That may be “telling” to you but I’m not sure why that should cast a negative light on Keras, really.
- zone411 4y agoI didn't say that it casts a negative light on Keras. Just on its popularity among outsiders. There are thousands of great libraries out there that are much less popular than Keras or PyTorch. And BTW, JAX is a useful Google-created framework that's growing in popularity among researchers and pushed PyTorch to improve (functorch), so I have nothing against Google projects.
- _ntka 4y agoThis is not true, the initial Keras port of the model was done by Divam Gupta who is not affiliated with Keras or Google. He works at Meta. The benchmark in the article uses mixed precision (and equivalent generation settings) for both implementations, it's a fair benchmark. In the latest StackOverflow global developer survey, TensorFlow had 50% more users than PyTorch.
- zone411 4y agoTwo Keras creators are listed as authors on this post. If they were not involved, this should be specified. I specifically talked about research and StackOverflow is not in any way representative of what's used. Do you disagree that the majority of neural net research papers now only have PyTorch implementations, not TensorFlow? Also, according to Google Trends, PyTorch is more popular: https://trends.google.com/trends/explore?geo=US&q=pytorch,tensorflow,keras https://trends.google.com/trends/explore?geo=US&q=pytorch,te.... BTW, I would love it if TF made a strong comeback, it's always better to have two big competing frameworks and I have some issues with PyTorch, including with its performance.
- polygamous_bat 4y ago> In the latest StackOverflow global developer survey, TensorFlow had 50% more users than PyTorch. It also doesn't help that PyTorch has its own discussion forum [1] where most pytorch questions end up. [1]: https://discuss.pytorch.org/ https://discuss.pytorch.org/
- nextaccountic 4y agoIs this faster even after applying the optimizations that reduce VRAM usage? (some of which the Keras version seem to lack)
- labarilem 4y agoVery interesting performance. Also a very good write-up. Can't wait to try this.
- gpderetta 4y agoI have a mediocre GPU but a fast CPU (with a lot of RAM). Would I see improvements there? I guess I should give it a try.
- senthilnayagam 4y agotried it yesterday, on intel i9 macbook pro it takes about 300 seconds per image.
- gpderetta 4y agoYou mean the keras version? How does it compare to the original one? Currently on my 10850k I get 2.4s/iteration, which is borderline usable. I haven't managed (nor tried very hard) to get the cuda version working on my 1070; I expect to be a little better, but I don't want to fight with ram issues.
- ttflee 4y agoHow many steps did you perform? I tried some and found no major differences after 16 steps or so with given random seed.
- ttflee 4y agoOn intel MacBookPro 2020, CPU-only, the original one[1] using pytorch utilized one core only. A tensorflow implementation[2] with oneDNN support which utilized most of the cores ran at ~11sec/iteration. Another OpenVINO based implementation[3] ran at ~6.0sec/iteration. [1] https://github.com/CompVis/stable-diffusion/ https://github.com/CompVis/stable-diffusion/ [2] https://github.com/divamgupta/stable-diffusion-tensorflow/ https://github.com/divamgupta/stable-diffusion-tensorflow/ [3] https://github.com/bes-dev/stable_diffusion.openvino/ https://github.com/bes-dev/stable_diffusion.openvino/
- gpderetta 4y agoYes, I use [3] and I get 2.4s/iter on my 10 core machine. I was wondering if keras would give additional help here. I'll have to try I guess.
- erwinh 4y agoNot necessarily my expertise but if as stated by the article, 2 lines of code can already get a 2x performance gain, what more can be done to improve performance in the coming years?
- londons_explore 4y agoIt's not two lines of code... It's 2 lines that enable tens of thousands of lines of library code by invoking a new optimizer...
- MintsJohn 4y agoI'm curious whether this really is "the fastest model yet" there are pytorch optimizations as well. Something like global optimization has been done in pytorch, here's a blog about it: https://www.photoroom.com/tech/stable-diffusion-25-percent-faster-and-save-seconds/ https://www.photoroom.com/tech/stable-diffusion-25-percent-f... Mixed precision seems pretty much default looking at a few Stable Diffusion notebooks. More intriguing, there's also a more local optimization that makes pytorch faster: https://www.photoroom.com/tech/stable-diffusion-100-percent-faster-with-memory-efficient-attention/ https://www.photoroom.com/tech/stable-diffusion-100-percent-... Unless it's already there, that last one would be interesting to add to keras. All in all this machine learning ecosystem is wild, as a software dev, things like cache locality and preferring computation over memory access are basic optimizations, yet in machine learning it seems wildly disregarded, I've seen models happily swapping between gpu and system memory to do numpy calculations. Hopefully stable diffusion changes things, the work towards optimizations is there, it just seems often disregarded. As stable diffusion is one popular open model that, when optimized, can be run locally (and not as saas, where you just add extra compute power, which seems cheaper than engineers) and has a lot of enthusiasm behind it, it might just be the spark that makes optimization sexy again.
- xiphias2 4y ago,, Note that when running on a M1 MacBookPro, you should not enable mixed precision, as it is not yet well supported by Apple's Metal runtime” It is a bit sad if this is just a closed software issue that cannot be fixed :(
- capableweb 4y agoIs it really that sad? Closed software/hardware won't get support (official nor community) for things until the maintainer of the software adds it, and people who buy that kind of hardware is more than aware of this pitfall (and in fact, see it as a benefit sometimes too).
- lynndotpy 4y agoI'm a new MacOS user and, while I did anticipate some of these issues, I do often find myself surprised when running into them. This was one such surprise I hit recently
- ribit 4y agoMixed precision won't do anything on Apple Silicon anyway since there is no performance advantage to using FP16 (aside from decreasing register pressure and RAM bandwidth which won't happen here as data is FP32 to start with).
- mrtksn 4y agoOn a 16GB 8c8g Macbook Air M1, the PyTorch implementation takes about 3.6s/step which is about 3 minutes per image with the default parameters. I wonder how faster this would be. If there's anyone out there with a similar system and wants to compare, could you please write your findings?
- thisisjasononhn 4y agoNot M1 comparible but I'm working on testing various GPU vs M1 comparisons, with a few accessible cloud providers. My impression is times should be the same, but it's nice to hear other real-world stats for M1 with SD. Makes me really want to rent the Hetzner M1 now. Which repo or build are you using BTW, is it the one related to this readme? https://github.com/magnusviri/stable-diffusion/blob/main/README-Mac-MPS.md https://github.com/magnusviri/stable-diffusion/blob/main/REA...
- mrtksn 4y ago>Which repo or build are you using BTW, is it the one related to this readme? https://github.com/magnusviri/stable-diffusion/blob/main/REA https://github.com/magnusviri/stable-diffusion/blob/main/REA... Yes, this one. However it was like a month ago I think, so speeds might have improved. I'm getting ~2.2s/step with another implementation: https://news.ycombinator.com/item?id=33006447 https://news.ycombinator.com/item?id=33006447
- thisisjasononhn 4y agoWow, that sounds like a good improvement. I am also wondering, do you follow the general advice of 1 iteration and 1 sample, for example: --n_samples 1 --n_iter 1 (when referencing commands using txt2img.py) I figure you could wait a bit for things to process going further, but curious just if you're getting results like that with higher sample/iter settings.
- mrtksn 4y agoI usually go with the default parameters.
- WatchDog 4y agoI don't quite understand the benefit of mixed precision. It seems like using high precision is useful for training, but if not training, why not just use float16 weights and save the memory?
- NavinF 4y agoConverting weights to float16 after training will reduce quality/accuracy whereas mixed precision has a negligible effect on quality/accuracy and dramatically improves performance. If you really just want to save memory, there's plenty of other low hanging fruit. It's just not a priority for most devs since mid tier GPUs start at 10GB whereas a typical model only has 0.5GB weights. Activations and intermediate calculations use way more memory.
- zone411 4y agoYou usually can. But it can take some work if you're using any libraries that expect FP32 and it might be slower, depending on the GPU. The FP16 support isn't quite as good as FP32.
- capableweb 4y agoTried to get this running on my 2080ti (11GB VRAM) but hitting OOM issues. So while performance seems better (but can't actually test this myself), I'm unable to actually verify it as it doesn't run. Some of the Pytorch forks works on as little as 6GB of VRAM (or maybe even 4GB?), but always good to have implementations that optimize for various factors, this one seems to trade memory usage for raw generation speed. Edit: there seems to be a more "full" version of the same work available here, made by one of the authors of the submission article: https://github.com/divamgupta/stable-diffusion-tensorflow https://github.com/divamgupta/stable-diffusion-tensorflow
- Karuma 4y agoThere are forks that even work on 1.8 of VRAM! They work great on my GTX 1050 2GB. This is by far the most popular and active right now: https://github.com/AUTOMATIC1111/stable-diffusion-webui https://github.com/AUTOMATIC1111/stable-diffusion-webui
- jaggs 4y agoThis needs Windows 10/11 though?
- Karuma 4y agoNope. There are instructions for Windows, Linux and Apple Silicon in the readme: https://github.com/AUTOMATIC1111/stable-diffusion-webui https://github.com/AUTOMATIC1111/stable-diffusion-webui There's also this fork of AUTOMATIC1111's fork, which also has a Colab notebook ready to run, and it's way, way faster than the KerasCV version: https://github.com/TheLastBen/fast-stable-diffusion https://github.com/TheLastBen/fast-stable-diffusion (It also has many, many more options and some nice, user-friendly GUIs. It's the best version for Google Colab!)
- jaggs 4y agoBrilliant thanks.
- deleted 4y ago
- mikereen 4y agoenhance
- nextaccountic 4y agoDoes this run on AMD? A problem I see is that a lot of times everything works fine on rocm+hip, but since nvidia dominates the machine learning market (and thus most researches run nvidia), most forks don't bother checking and just advertise compatibility with nvidia and sometimes apple M1. Problem is, AMD GPUs are much cheaper!
- cypress66 4y ago> Problem is, AMD GPUs are much cheaper! Are they? I believe Nvidia (consumer) gpus have better price/performance than amd for AI.
- nextaccountic 4y agoI don't know about AI performance (does this happen only because of the overhead of providing CUDA through rocm+HIP?), but I was just checking and at least in my country (Brazil), for any given memory size (12GB, 8GB, 4GB) I can find cheaper AMD GPUs than NVidia GPUs Here I'm considering that the main constraint is VRAM and while stable diffusion now runs even on GPUs with 2GB RAM, there's always new developments that require more VRAM (for example, Dreambooth requires 12GB as of today)
- mrtranscendence 4y agoMaybe for AI? For other tasks, especially gaming, they punch well above their weight relative to Nvidia (though they lack features in comparison). It's also possible to get a 16GB card for much cheaper from AMD than Nvidia.
- mrtksn 4y agoWell, high-end stuff is always on Nvidia and Apple Silicon seems to get some love because of its unified memory that makes it possible in first place plus its popularity among developers. AMD seems to be popular among gamers on budget and the budget cards often don't have the VRAM required by default. So, AMD seems to be in this weird place where the people who can make it work don't care.
- 4y ago
- unspecldn 4y agoHow do I deploy this? Can someone offer some guidance please?
- JoeAltmaier 4y agoThe otter examples highlight something you can't control using these things: the 'eats shoots and leaves' phenomenon. The prompt was "A cute otter in a rainbow whirlpool holding shells, watercolor" Seems like the otter should be holding shells, the way a normal human parses it. The tool showed the otter 'in holding-shells', which are shells that hold otters apparently. Also some random shells strewn about, as the technique is sensitive to spurious detail sprouting up from single words. Until the tool permits some kind of syntactic diagramming or so forth, we'll not be able to control for this. Just the other day here, I saw a picture of a fork and some plastic mushrooms. The prompt was 'plastic eating mushrooms' which was ambiguous even to humans. The tool chose to illustrate the subclass of mushrooms 'eating-mushrooms' (as opposed to poison mushrooms or decorative mushrooms I suppose) made of plastic. When we're playing around this can seem whimsical and artistic. But a graphic designer might want some semblance of control over the process. Not sure how a solution would work.
- CuriouslyC 4y agoGraphic designers lean on img2img in their workflows more than txt2img, as that gives you the control you speak of.
- rytill 4y agoDo you know of examples of graphic designers who have shared their img2img workflows online?
- CuriouslyC 4y agoHang out in reddit.com/r/stablediffusion, people have posted workflows quite a bit.
- CrazyStat 4y agoThis is the compositionality problem--the language model sometimes doesn't quite know how to put the words together. Better language models will help in the future; in the mean time you can give it a helping hand by prompt engineering or using img2img.
- monkmartinez 4y agoIs the H5 file type that much different than whatever the Pytorch versions are using? The model is loaded from Huggingface during the instantiation of the stable diffusion class. It is loaded as an H5 file which I believe is unique to Keras[0]. I don't have any experience with Keras so I can't say if that is good or bad. I wanted to see where they were getting the weights as the blog post didn't demonstrate an explicit loading function/call like Pytorch. Gonna run it and see... although I have like 40GB of stable diffusion weights on my computer now. [0] https://github.com/keras-team/keras-cv/blob/master/keras_cv/models/generative/stable_diffusion/stable_diffusion.py https://github.com/keras-team/keras-cv/blob/master/keras_cv/...
- jawadch93 4y ago[dead]
- shadowgovt 4y agoBonus points for this article being one of the clearest explanations for how Stable Diffusion works that I've seen to-date.
- mpaepper 4y agoHere a bit more in depth how it works: https://www.paepper.com/blog/posts/how-and-why-stable-diffusion-works-for-text-to-image-generation/ https://www.paepper.com/blog/posts/how-and-why-stable-diffus...
- lordswork 4y agoNot familiar with the other authors, but François Chollet (author of Deep Learning with Python) is truly gifted in the art of pedagogy.
- gdubs 4y agoHas anyone tried running this with an AMD card on Mac? At first glance it's able to run on Metal (given the M1 compatibility)...
- dennisy 4y agoThis is amazing! I am more used to TF so very happy to see this! Has anyone got a suggestion on how to fine tune this model?
- honksillet 4y agoCan this be used to train you own model? I have a moderately large medical image dataset that would like to try this with for data augmentation.
- technerder 4y agoI've been seeing a lot of HN submissions and images generated by stable diffusion but I've yet to actually toy around with it due to a lack of time, what fork appears to produce the best results for people that don't have a 3090s worth of VRAM? (Currently only have a 3060 with 12GB which I thought would be enough for almost anything haha)