9 ms·
SDXL Turbo: A Real-Time Text-to-Image Generation Model
- minimaxir 3y agoThe license is non-commercial, but: > For clarity, Derivative Works do not include the output of any Model. https://huggingface.co/stabilityai/sdxl-turbo/blob/main/LICENSE.TXT https://huggingface.co/stabilityai/sdxl-turbo/blob/main/LICE... Doesn't that mean that generated images from it should be fine for commercial?
- nperez 3y agoI'm not a lawyer, but I believe there was a court ruling that says AI generated works cannot be copyrighted. So you could use them, but couldn't stop anyone else from doing what they want with them
- yreg 3y ago> there was a court ruling What jurisdiction? USA?
- adventured 3y agoYes - https://www.reuters.com/legal/ai-generated-art-cannot-receive-copyrights-us-court-says-2023-08-21/ https://www.reuters.com/legal/ai-generated-art-cannot-receiv...
- donkeyd 3y agoActually, no. This is just another example of a headline leaving out important details of the actual case. In this case the plaintiff actually named the AI as the producer, not themselves. From the case: "the sole issue of whether a work generated entirely by an artificial system absent human involvement" This leaves a lot of wiggle room for AI created art with some form of human involvement. I.E. import the generated image into Photoshop and edit it. Perform some inpainting to improve certain parts. Possibly even prompting and configuring things in Automatic1111 might be regarded as human involvement. This is going to bring many legal procedures before there's a clear answer to whether AI generated art can be copyrighted.
- fragmede 3y agoit doesn't touch trademarks either. If I generate a recognizable picture of a trademarked icon, I still can't use it for my own commercial purposes, even if it's not covered under copyright.
- kmeisthax 3y agoNo, it just means that if they sue you, they're pre-committing to not try and foreclose on your own generated outputs by claiming they're derivatives that they would then own. Of course, this is a water sandwich. If model outputs are derivatives of the model, it'd be difficult to argue that the model itself isn't a derivative of all the training data, most of which isn't licensed. So if anything, this covers Stability's ass, not yours. There's also the related question of if AI models - not their outputs, just the models themselves - have any copyright at all. The logic behind the non-copyrightability of AI art would also apply to the AI training process, so the only way you could get copyright would be a particularly creative way of organizing and compiling the training dataset. Remember: while the "AI art isn't art" argument reeks to high heavens of artistic snobbery, it's not entirely wrong. There isn't a lot of creative control in the process. Furthermore, we don't give copyright to monkeys[2], so why should we give it to AI models? "Noncommercial" isn't actually a thing in copyright law. Copyrighted works are inherently commercial artifacts[0], so if you just say "noncommercial use is fine", you've said nothing - and you've invited the legal equivalent of nasal goblins[1] into the courtroom. Creative Commons gets around this by defining their own concept of NonCommercial use. So what did Stability's lawyers cook up? > “Non-Commercial Uses” means exercising any of the rights granted herein for the purpose of research or non-commercial purposes. Non-Commercial Uses does not include any production use of the Software Products or any Derivative Works. Uh... yeah. That's replacing a meaningless phrase with a tautology. Fun. The only concrete grant of rights is research use, and they categorically reject "any production use", which is awfully close to all uses. Even using this to generate funny fanart mashups for your own personal enjoyment could be construed as a 'production use'. Stability could actually sue you for that (however unlikely that would be). [0] In the eyes of the law. I actually hate this opinion, but it's the opinion the law takes. [1] Under ISO 9899, it is entirely legal for C programs with undefined behavior to make goblins fly out of your nose. [2] https://en.wikipedia.org/wiki/Monkey_selfie_copyright_dispute https://en.wikipedia.org/wiki/Monkey_selfie_copyright_disput...
- __loam 3y agoI think the artists get to say whatever they want about this bullshit considering it's entirely dependent on their work to even function. I don't think the AI community gets to call the people who make the models possible at all snobs or anything else. The AI people didn't make shit. They should have some respect for the people that do.
- esjeon 3y agoIANAL but it sounds more like SAI is not responsible for any outputs generated using their models and how those are used.
- rkwasny 3y agoAwesome! so where is the API I can try?
- dreadlordbone 3y agoThey have a web demo here: https://clipdrop.co/stable-diffusion-turbo https://clipdrop.co/stable-diffusion-turbo
- KennyBlanken 3y ago...which requires you to sign in. That nice little text box invites you until you actually click to enter some text and get a registration box thrust at you People that design a UX where the user tricked into a registration 'ambush' need to be punched in the face.
- LordDragonfang 3y agoLuckily this one accepts burner emails just fine, without any intrusive other data collection (name, etc) http://grr.la http://grr.la
- fragmede 3y agoThat's a useful website, but holy shit it is utterly unusable without an adblocker.
- LordDragonfang 3y agoI mean, you should have an adblocker on every device anyway. On my phone I use https://adguard-dns.io/en/public-dns.html https://adguard-dns.io/en/public-dns.html (No need to download the app, just select "Configure AdGuard DNS manually"). As a bonus, on android, it blocks ads in shitty mobile games too.
- meeech 3y agoA totally reasonable response to a minor inconvenience.
- LordDragonfang 3y agoBased on the demo, that's... incredibly fast. Literally generating images faster than I can type a prompt. They've clearly got a set seed, so they're probably caching request, but even with prompts that they couldn't possibly have cached it's within a second or so.
- gremlinsinc 3y agoI hope playgroundai.com adopts this asap, but not sure they can with that non-commercial bit...
- deleted 3y ago[deleted]
- stets 3y agoThis isn't cached. I'm running it locally on a 4060 TI 16 GB and it's just as fast. Image gens in .6-.8 seconds. Each word or character I type is a new image gen and it's INSTANT.
- thot_experiment 3y agoI've been mucking with this stuff again and the SDXL + LCM sampling & LoRA makes 1280x800 images in like 2 second, so about a ~5x speed increase for me (so this would be roughly 2x faster than LCM (??, napkin math)). I've found that the method isn't as good at complex prompts. They claim here this can outperform SDXL 1.0 WRT prompt alignment, but I'm curious what their test methodology is. I searched the paper and I couldn't immediately find how it was evaluated. I think these sorts of subjective measurements are fiendishly hard to quantify given the infinity of possible prompts. Still, exciting stuff always happening here, what a time to be alive.
- htrp 3y agothe use case here is really for segmented inpainting. don't like a part of an image? replace it instantly
- thot_experiment 3y agoThere are SO many use cases! I maintain we're not even scratching the surface here. You could programmatically reskin video based on crowd participation, you could re-texture VR spaces on the fly. The space of cool shit that you can do with this stuff is growing far faster than we're able to explore it right now.
- __loam 3y ago[flagged]
- refulgentis 3y ago?
- __loam 3y ago[flagged]
- 3y ago
- xena 3y agoI'm gonna wait until this is released as a model. This is really cool though!
- SushiHippie 3y agoFrom TFA: > Download the model weights and code on Hugging Face[0], currently being released under a non-commercial research license that permits personal, non-commercial use. [0] https://huggingface.co/stabilityai/sdxl-turbo https://huggingface.co/stabilityai/sdxl-turbo
- xena 3y agoI am fucking blind, thanks!
- lemoncookiechip 3y agothe clipdrop demo doesn't inspire much confidence with how bad the generations are, we're talking 2021 levels, not to mention everything is NSFW somehow, should probably work on those filters.
- minimaxir 3y agoI tested a bit and the quality for photorealistic images is surprisingly bad, and definitely worse than LCM and of course normal SDXL. For more artistic images, SDXL Turbo fares better. Unlike normal SDXL, you're required here to use the old-fashioned syntatic sugar like "8k hd" and "hyperrealistic" to align things.
- jauntywundrkind 3y agoAre there any good resources for learning a lot of "syntactic sugar" terms? This is new to me, but I'd love to know more.
- swyx 3y agohttps://github.com/swyxio/ai-notes/blob/main/IMAGE_PROMPTS.md https://github.com/swyxio/ai-notes/blob/main/IMAGE_PROMPTS.m...
- jauntywundrkind 3y ago#LearningInPublic strikes again! I love you swyx!
- swyx 3y agohaha thank you ser. best way to thank me is to start your own LIP practice and pass it on :)
- genewitch 3y agoIt is completely dependent on the model. Civit dot ai has model showcases as well as fine-tune showcases, and you can click any image or press the (i) to see the generation info. Some models like natural language prompts - "draw me a pterodactyl tanning at a beach", some prefer shorthand (danbooru style clip) - "1man, professor, classroom, chalkboard, white_hair, suit", and some work with a mixture of the above as well as the syntactical sugar -"masterpiece, 8k, trending on artstation, space image, a man floating next to a spaceship in space, bokeh, rim lighting, cinematic lighting, Nikon D60, f / 2" Fine-tuning models - LoRA, etc, allow one to convert prompts from one style to another if they wish, but usually it's to compress an idea, style, person, object, etc in to a single "token", so you can work on other aspects of the image. Check out civit AI and you can sort of get an idea of the cargo cultism as well as what sort of keywords actually make a difference.
- minimaxir 3y agoHugging Face released a Colab Notebook for generation from SDXL Turbo using the diffusers library: https://colab.research.google.com/drive/1yRC3Z2bWQOeM4z0FeJ0rF6fnDTTSdnAJ https://colab.research.google.com/drive/1yRC3Z2bWQOeM4z0FeJ0... Playing around with the generation params a bit, Colab's T4 GPU can batch-generate up to 6 images at a time at roughly the same speed as one.
- 3abiton 3y agoSo the new SD model requires higher end hardware compare to the rest?
- minimaxir 3y agoNo, it's a smaller model than normal SDXL so it requires less hardware compared to SDXL.
- albert_e 3y agothanks for sharing I got to experience the power of current models with just 5 lines of code the pace of change is stressing me out :)
- sp332 3y agoIs Euler Ancestral recommended for this? I thought the Ancestral samplers added noise to every step, preventing convergence.
- webmaven 3y agoDoes Turbo (the model, not this particular notebook) support negative prompts?
- smusamashah 3y agoSDXL is already very very slow when compared to SD 1.5. They are claiming 200ms for 512x512 image in SDXL on A100. We need SD 1.5 turbo for even faster generation.
- ShamelessC 3y agoIt _would_ be nice if they offered 512/768/1024 px variants of the models. I frequently don’t actually need the full 1024 px as it just needs to look good enough for a chat thumbnail. Then I could upscale it manually later. There’s other models like Kandinsky but it’s not super convenient to use multiple models with different code and what not.
- Sharlin 3y agoI haven't found SDXL to be inherently much slower than 1.5, besides the obvious 4x slowdown from having twice the linear resolution.
- thot_experiment 3y agoYeah, in my experience it's actually FASTER because I was already doing high res gens, and that required 2 or 3 passes previously.
- minimaxir 3y agoSDXL Turbo also uses a distilled version of SDXL so it gets a speed bonus from that too.
- swyx 3y agois it known how much larger SDXL is compared to SD1 or SD2?
- dragonwriter 3y agoSD1.x is around 1.1B parameters including VAE, SD2.x is slightly more (uses the same UNet and VAE, but a bigger text encoder; not finding stats as quickly as I'd like), and SDXL is 3.5B parameters (single model only, but StabilityAI's preferred base + refiner model setup is effectively 6.6B parameters -- some bits are shared.)
- ilaksh 3y agoDoes anyone have any idea of when/if there will be a simple way to get commercial access in one step? Like an API or something? Or if they want to charge for it, then maybe a web check out? It's interesting that they finally decided to try to make something commercially restricted. Do they have a watermark or anything that they can use to track down people who use without a license? Also, is there anything like an open source (commercial allowed) effort at reproducing this?
- zaptrem 3y agoOpen version predates this one: https://huggingface.co/blog/lcm_lora https://huggingface.co/blog/lcm_lora
- ilaksh 3y agoYes but Turbo is 4 times faster.
- fernly 3y agoabout my 8th prompt, it told me I had used up my usage and offered an option to sign up for "pro".
- ilaksh 3y agoMe too, but I need an API.
- liuliu 3y agoInstruction here for how to use it on iPhone / iPad / Mac today: https://twitter.com/drawthingsapp/status/1729633231526404400 https://twitter.com/drawthingsapp/status/1729633231526404400 (note that you need to convert 8-bit version yourself if you want to use it on < 8GiB devices).
- donkeyd 3y agoI remember when you shared the first version on HN, I was totally impressed with what my little phone could do. It was my first step of many into the incredible world of SD. I never expected this app to be maintained this long and especially this actively. And all of this for free. So thank you very much for your work! PS, will you be adding Turbo directly to the app in the near future?
- liuliu 3y ago> PS, will you be adding Turbo directly to the app in the near future? I need to get some clarity from Stability given the new Membership thing. I personally see no reason why I cannot as long as I tell people these weights are non-commercial / academic only.
- Severian 3y agoWorks with Automatic111. Generated 20 512x512 on a lowly RTS 2070S with 8GB RAM. Prompt: a man Steps: 1, Sampler: Euler a, CFG scale: 1, Seed: -1, Size: 512x512, Model hash: e869ac7d69, Model: sd_xl_turbo_1.0_fp16, Clip skip: 2, RNG: NV, Version: v1.6.0 Examples: https://imgur.com/a/UuuT9qu https://imgur.com/a/UuuT9qu
- techbro92 3y agoI just want to point out that I’ve noticed sdxl isn’t good at producing images that are 512x512 for some reason. It works much better with at least 768x768 resolution.
- minimaxir 3y agoNormal SDXL requires 1024x1024 output or the quality degrades significantly.
- doctorhandshake 3y agoHi Max! Thanks for all the tuts! Small correction - SDXL wants ~1 megapixel resolutions at a variety of aspect ratios. https://github.com/lllyasviel/Fooocus/issues/24 https://github.com/lllyasviel/Fooocus/issues/24
- minimaxir 3y agoFair point, although in my experience SDXL still isn't great at non-square ratios. I end up just cropping them to the ratio I want.
- doctorhandshake 3y agoWhat have you seen to be the issue? Composition? Realism? Prompt adherence? I’m just finishing a project having generated tons of images at a mix of ~6 aspect ratios and I haven’t noticed any difference.
- 3y ago
- Filligree 3y agoIt's... fast, yeah, but the quality is low. Not sure what I'd be using this for.
- smcleod 3y agoVery impressive, I'm generating some really nice quality images at 1024x1024 in under 1 second on a 3090 in InvokeAI!
- kmeisthax 3y agoNoncommercial use - aside from being one of my licensing pet peeves - seems to indicate that the money is drying up. My guess is that the investors over at Stability are tired of subsidizing the part of the generative AI market that OpenAI refuses to touch[0]. The thing is, I'm not entirely sure there's a paying portion of the market? Yes, I've heard of people paying for ChatGPT because it answers programming questions really well, but that's individual users, who are cost sensitive. The real money was supposed to be selling these things as worker replacement, but Hollywood unions have (rightfully) shut AI companies out of the markets where a machine that can write endless slop might have made lots of money. OpenAI can remain in Microsoft's orbit for as long as the stupid altruist / accelerationist doomer debate doesn't tear them apart[1]. Google has at least a few more years of monopoly money before either the US breaks them up or the EU bankrupts them with fees. I don't know who the hell is pumping more money into either Anthropic or Stability. [0] Porn. It's always porn. OpenAI doesn't want to touch it for very obvious reasons. [1] For what it's worth, Microsoft has shown that all the AI safety guardrails can be ripped out by the money people at a moment's notice, given how quickly they were able to make the OpenAI board blink.
- fnordpiglet 3y agoOpen AI has a pretty robust and profitable business without Microsoft. In every enterprise I’ve been involved with over the last few years we have had some incredibly material and important use cases of OpenAI LLMs (as well as Claude). They aren’t spewing slop or whatever, they’re genuinely achieving valuable and foundational business outcomes. I’ve been a bit stunned at how fast we’ve achieved these things and it tells me that the AI hype isn’t hype, and that if we have done these things in a year, it’s hard to estimate how much impact the technologies will have in five but I think it’s substantial. So is our spend with OpenAI. Or rather, with Azure on OpenAI products. The only value from our experiences Microsoft offers is IAM - which is sufficient frankly. Stability and Midjourney are also making money, but it’s largely with amateurs and people prototyping content for their own creations. A lot of single person Indy game developers are using these tools to generate assets, or at minimum first pass assets. I think a lot of media companies are producing the art for their articles or news letters etc using these tools. Whether this is enough, I don’t know.
- godelski 3y ago> A Real-Time Text-to-Image Generation Model > On an A100, SDXL Turbo generates a 512x512 image in 207ms (prompt encoding + a single denoising step + decoding, fp16), where 67ms are accounted for by a single UNet forward evaluation. Okay... so what part of this is real time? 207ms is 4.8Hz. 67ms is 14.9Hz. Isn't "real time" in graphics considered to be at least 30Hz (33ms)? And by today's standards at minimum 60Hz (16ms) if not 144Hz (7ms)? I'm lost at what part of this is real time? I'm not sure it even would get there with an H100. Maybe an H100 and everything is TensorRT?
- LelouBil 3y agoIt's "real-time" as in "I finished typing and the image is already there"
- godelski 3y agoI think it is still a but deceptive to say this considering a well know and high usage alternative definition exists. Plus that's on commercial grade hardware.
- stets 3y agoConsidering the use-case of just interacting with a computer and typing prompts, I'd call it real-time.
- iAkashPaul 3y agoFrom testing out the 1024px outputs it seems to work okay for text2img tasks only, other tasks get deepfried results. Also got it working using SDXL-LCM with similar results.
- DeathArrow 3y agoThat's really great, we can generate porn faster and better. :)
- joelthelion 3y agoWhat are the use cases for this?
- bulbosaur123 3y agoIs this available in Automatic111?