5 ms·

I Resurrected “Ugly Sonic” with Stable Diffusion Textual Inversion

289 points by minimaxir 4y ago

beeskneecaps 4y ago
Ah the things mankind can achieve.
- version_five 4y ago
  I'm eager to see if "AI" can break out of these local minima and actually live up to its promise, or if the end game is really just making cool looking pictures that get fleeting internet attention
- fny 4y ago
  It's beautiful. Just imagine what'll be possible if we learn to manipulate meatspace like this.
searchableguy 4y ago
Yeah, textual inversion is amazing paired with SD. I recently trained it on NFTs to generate variants of bored ape and punk style art. The only problem right now is the variants are not consistent and it's hard to tell stable diffusion to make only slight variant changes with some mask and editing. You can mask out the top head of the NFT punk and stablediffusion can generate different heads but that's fairly limited in the end result. I think a cloud service which can automate the training and store textual inversion models would be a really cool startup. Ping me if you want to build this together. I've been experimenting with it to spit out commercial style illustrations and stock photos. It's a lot of manual work with Google collab and frustrating to try.
- deleted 4y ago
  [deleted]
- deleted 4y ago
  [deleted]
msoucy 4y ago
Your scientists were so occupied with whether they could...
bjt2n3904 4y ago
I've always been bothered by app names getting invaded by keyword vomit. "Firefox, fast and free" It feels like going to get a movie, and the title is "Casablanca, super attractive lead actor" or something. It just cheapens the value of the work like it's a low budget film from the adult section. But what weirds me out is how much of that is needed to get a desired result. "Unreal engine 4k resolution". What happens when 4k isn't enough? What about twenty years later when all our buzzwords are meaningless, and unreal engine no longer exists? Or what if AI appropriates the word, and that becomes the new definition?
- minimaxir 4y ago
  The prompt keywords are better thought of as "cheat codes" where the AI knows that those keywords strongly correlate with good imagery.
  - deleted 4y ago
    [deleted]
- l33tman 4y ago
  It hasn't got anything to do with the keywords actual meaning really... all of this is just to map the 7000-dimensional internal description space in SD to something we meagre humans can input and output. You can leave it free, but then you have to accept the result can be anything. Normally you do want to leverage some control of the process. The keyword "4k" doesn't "mean" anything to the model, it maps to the internal space through the language model depending on its training on captions earlier. You could just as well have used any other way of assigning some parts of this 7000-dimensional vector, it just turns out these keywords are usable shortcut, just like specifying a camera model or an artist name in essence acts as a "macro" to the internal space.
  - bjt2n3904 4y ago
    Right right. It just seems like a way that could evolve language in a dystopian way. If 4k doesn't really "mean" anything to the algorithm, will it still have meaning for us? Or will our usage of the word start to reflect how the algorithm interprets it? Just like YouTube face. It starts out as us influencing the algorithms, it ends with the algorithms influencing us.
devindotcom 4y ago
Good stuff, Max. Wish I had time to fiddle with local SD but I'm not worried about it, it just keeps getting better and more accessible. The leading edge will always be experimenters but the trailing edge will be creative people using the tools to further their art.
nullc 4y ago
Anyone know a TI training repo that's a bit more memory optimized than the original one? It's a bit kludgy getting it working on a 12 GB gpu, I managed it but I'd rather a less hacky solution.
- f38zf5vdt 4y ago
  https://github.com/lstein/stable-diffusion https://github.com/lstein/stable-diffusion Works with M1 Macs too.
shadowgovt 4y ago
> I’ve always had difficulty generating a normal Sonic the Hedgehog image with AI image generation. DALL-E 2, for example, just flat-out can’t do it. IIUC, OpenAI specifically deny-listed trademarked characters from the training dataset to try and side-step getting sued by companies with enough resources to move the needle on what was allowed and disallowed in terms of digital art creation (since a ruling could disrupt their entire business model). I assume Emad Mostaque did the risk calculus differently when they open-sourced Stable Diffusion (I don't know for sure, but it smells like his attitude on the question was "I don't care because now that it's open-source nobody can delete it from the Internet anyway").
- minimaxir 4y ago
  That's true, although I'd still expect something a little blue. I had similar issues with VQGAN + CLIP, where CLIP is also trained by OpenAI but otherwise handles copyrighted characters a bit better.
  - shadowgovt 4y ago
    If Sonic the Hedgehog was specifically excluded from the dataset, DALL-E 2 has no way to know the hedgehog should be blue. Instead, I think it's interpreting "a portrait of Sonic the Hedgehog" as a hedgehog that happens to have a name, and almost all hedgehogs, regardless of their name, are the same color. Even if someone had added a blue realistic-looking hedgehog to the dataset, if they referenced Sonic their image would have been deny-listed. So the only way to get there is to add the adjectives one is looking for, such as `a portrait of a blue hedgehog wearing red sneakers and eating a chili dog`... And, indeed, when I try that prompt, I definitely get some Sonic-alikes.
- mg 4y ago
  Dall-E 2 can render Sonic the Hedgehog just fine if you give it a bit more context: https://twitter.com/illubots/status/1571921642569502720 https://twitter.com/illubots/status/1571921642569502720
elteto 4y ago
Quick question: what is it with tagging these SD prompts with “trending on artstation”? Why is specifically artstation important?
- beschizza 4y ago
  A known dataset source with a strong quality filter
- faldore 4y ago
  and Greg Rutkowski too
  - minimaxir 4y ago
    To add on this: I specifically avoided using him (or other living artists) in the prompts for this post since that's been a hot issue lately: https://www.technologyreview.com/2022/09/16/1059598/this-artist-is-dominating-ai-generated-art-and-hes-not-happy-about-it/ https://www.technologyreview.com/2022/09/16/1059598/this-art... I did test it though, which resulted in...this: https://twitter.com/minimaxir/status/1571520043737042946 https://twitter.com/minimaxir/status/1571520043737042946
- PeterisP 4y ago
  It turned out to be a simple way to communicate to the model "I want this to be similar to high quality images only, instead of all kinds of shitty art you've seen".
macrolime 4y ago
Gonna have to pick up one of those cheap RTX 3090s that miners are getting rid of so I can try this myself. With 8GB I'm just getting out of memory errors.
- 14 4y ago
  I was hoping an online service would pop up where you can use someone else's hardware for a small fee and generate images. I really want to give it a try but only have a gtx970 so don't think it will work.
  - kcartlidge 4y ago
    > I was hoping an online service would pop up where you can use someone else's hardware for a small fee and generate images. This one charges but gives pretty good results at 512x512 - and in only a couple of seconds at that resolution. For more logo/cartoony stuff you can generally get a 10 step done in less than a second. When I signed up a few weeks ago they gave 200 credits free (IIRC). After that it is $10 for another 1,000 credits (a credit gives 5 images at 10 step 512x512 or 1 image at 50 step 512x512). https://beta.dreamstudio.ai https://beta.dreamstudio.ai Only thing to note is that after a while it slows down, but I realised that is because every image generated is going into an array in local storage in the browser. Using the browser's Inspect -> Application area to clear that array every 100 images or so sorted that out.
  - pbhjpbhj 4y ago
    Places remote computer providers like Vultr, Digital Ocean, Linode, etc., have servers with GPUs you can run up. You pay for uptime, and storage (including machines that are not up).
- montebicyclelo 4y ago
  You can 100% run this on an 8gb card. Make sure to load the model weights in half precision (and optionally enable attention slicing). E.g., using the diffusers library [1]: from diffusers import StableDiffusionPipeline from torch import autocast pipe = StableDiffusionPipeline.from_pretrained( "CompVis/stable-diffusion-v1-4", revision="fp16", torch_dtype=torch.float16, use_auth_token=True ) pipe = pipe.to("cuda") pipe.enable_attention_slicing() prompt = "a photo of an astronaut riding a horse on mars" with autocast("cuda"): image = pipe(prompt).images[0] [1] https://github.com/huggingface/diffusers https://github.com/huggingface/diffusers
sophrocyne 4y ago
Stable Diffusion is wild - the space has been quickly developing and watching the pace of development makes me reconsider what I consider "staggering". I've been blown away. The accessibility of this technology is even more incredible - there's even a fork that is working on M1 Macs (https://github.com/lstein/stable-diffusion https://github.com/lstein/stable-diffusion) We are in for some interesting times. Whatever the next iteration of Textual Inversion is will be extremely disruptive, especially if the concepts continue to be developed collectively.
14 4y ago
Is there an online service where I could use the features of Stable Diffusion to create images? I don't think my gtx970 meets the specs needed but I am a total noob and not sure. I have been dying to try it out however and would like to find a way to make a few images. If anyone knows of any services I would be thankful for a link. Thanks
- kaeruct 4y ago
  I like beta.dreamstudio.ai
wombat-man 4y ago
I kinda wish they'd release that version as a special feature. Probably not worth the effort though
dougmwne 4y ago
How many images do you need to get a good textual inversion? If you were to draw a custom character in a few poses, could you then generate infinite new styles and poses of that character? Or does it have to be something already common in the training dataset that you are just putting a very specific noun to?
- Filligree 4y ago
  Five or six is recommended.
  - skinner_ 4y ago
    If someone played around with textual inversion: what happens if I give it hundreds of images? Does it plateau, does it regress, and if it regresses, what is the issue exactly?
cercatrova 4y ago
Anyone else getting bored of generating images? I used Stable Diffusion even before the public release (there was a guide on 4chan with leaked weights) and I'd been using it for a few weeks. I just feel like I've generated all the images I want to for now, and I just don't have any more interesting concepts I want to explore anymore. When you have so much variety in images, I got acclimatized to them and they all became the same and uninteresting after a while. It's like a hedonic treadmill but for AI image generation. I assume this will have the most usage as a once in a while tool for artists to get inspired from, and also a tool for commercial usage such as in a Photoshop or Figma plugin for designers. The lay person who wants to generate images will get bored after a while.
- spywaregorilla 4y ago
  No particular reason to think this is suddenly going to make you want to be an artist
- illubots 4y ago
  The first human who discovered that they could draw something on a wall probably also got bored after a while. And thought "I assume this will have the most usage as a once in a while tool to decorate my cave with funny animals". But then drawing animals turned into writing. Turned into printing. Turned into emails and the web and the web 2.0 and now here we are, sending our thoughts across the world with the speed of light. Using software that we wrote to tell computers how to do this for us. It might be similar with AI driven image generation. That it evolves in ways that are currently hard to foresee. Maybe one day, our thoughts are translated in realtime into images and send across the galaxy with the speed of light. Or faster?
  - BizarroLand 4y ago
    How cool will it be when we can tell a program, "I want a fantasy story that continues on Lord of the Rings and details Pippin and Merry's second great and wonderful adventure after Frodo left middle earth" and get a 300 page story that is well-written and inventive and fun to read. I know the requirements for something like that are extraordinarily high, but I could see it in 30-40 years being viable.
bscphil 4y ago
> Indeed, there are many images of Sonic in the training dataset, however the generated images do not verbatim reproduce or otherwise plagiarize results from the training set above (I checked each one). This understanding of plagiarism reminds me of what students tell me when I ask them why they thought they could paraphrase an uncited Wikipedia article as their paper, right before I report them to the dean's office.
- minimaxir 4y ago
  Plagiarism in art is more complicated than plagiarism in a thesis paper, especially since art is a creative field defined by inspiration and iteration. I added that line because reproduction of the input dataset into an AI is a valid concern (such as reproducing a Getty Photos watermark), but that isn't happening here.
spywaregorilla 4y ago
That's an ugly sonic. It doesn't really look like "Ugly Sonic" though.
King-Aaron 4y ago
Just an aside, if you want to see Ugly Sonic resurrected, just watch the new Chip N Dale movie... It's actually an alright film, and the fact that they put ugly sonic in it was pretty amazing

289 points by minimaxir 4y ago

beeskneecaps 4y ago

Ah the things mankind can achieve.

version_five 4y ago
I'm eager to see if "AI" can break out of these local minima and actually live up to its promise, or if the end game is really just making cool looking pictures that get fleeting internet attention
fny 4y ago
It's beautiful. Just imagine what'll be possible if we learn to manipulate meatspace like this.

searchableguy 4y ago

Yeah, textual inversion is amazing paired with SD. I recently trained it on NFTs to generate variants of bored ape and punk style art. The only problem right now is the variants are not consistent and it's hard to tell stable diffusion to make only slight variant changes with some mask and editing. You can mask out the top head of the NFT punk and stablediffusion can generate different heads but that's fairly limited in the end result. I think a cloud service which can automate the training and store textual inversion models would be a really cool startup. Ping me if you want to build this together. I've been experimenting with it to spit out commercial style illustrations and stock photos. It's a lot of manual work with Google collab and frustrating to try.

deleted 4y ago
[deleted]
deleted 4y ago
[deleted]

msoucy 4y ago

Your scientists were so occupied with whether they could...

bjt2n3904 4y ago

I've always been bothered by app names getting invaded by keyword vomit. "Firefox, fast and free" It feels like going to get a movie, and the title is "Casablanca, super attractive lead actor" or something. It just cheapens the value of the work like it's a low budget film from the adult section. But what weirds me out is how much of that is needed to get a desired result. "Unreal engine 4k resolution". What happens when 4k isn't enough? What about twenty years later when all our buzzwords are meaningless, and unreal engine no longer exists? Or what if AI appropriates the word, and that becomes the new definition?

minimaxir 4y ago
The prompt keywords are better thought of as "cheat codes" where the AI knows that those keywords strongly correlate with good imagery.
- deleted 4y ago
  [deleted]
l33tman 4y ago
It hasn't got anything to do with the keywords actual meaning really... all of this is just to map the 7000-dimensional internal description space in SD to something we meagre humans can input and output. You can leave it free, but then you have to accept the result can be anything. Normally you do want to leverage some control of the process. The keyword "4k" doesn't "mean" anything to the model, it maps to the internal space through the language model depending on its training on captions earlier. You could just as well have used any other way of assigning some parts of this 7000-dimensional vector, it just turns out these keywords are usable shortcut, just like specifying a camera model or an artist name in essence acts as a "macro" to the internal space.
- bjt2n3904 4y ago
  Right right. It just seems like a way that could evolve language in a dystopian way. If 4k doesn't really "mean" anything to the algorithm, will it still have meaning for us? Or will our usage of the word start to reflect how the algorithm interprets it? Just like YouTube face. It starts out as us influencing the algorithms, it ends with the algorithms influencing us.

devindotcom 4y ago

Good stuff, Max. Wish I had time to fiddle with local SD but I'm not worried about it, it just keeps getting better and more accessible. The leading edge will always be experimenters but the trailing edge will be creative people using the tools to further their art.

nullc 4y ago

Anyone know a TI training repo that's a bit more memory optimized than the original one? It's a bit kludgy getting it working on a 12 GB gpu, I managed it but I'd rather a less hacky solution.

f38zf5vdt 4y ago
https://github.com/lstein/stable-diffusion https://github.com/lstein/stable-diffusion Works with M1 Macs too.

shadowgovt 4y ago

> I’ve always had difficulty generating a normal Sonic the Hedgehog image with AI image generation. DALL-E 2, for example, just flat-out can’t do it. IIUC, OpenAI specifically deny-listed trademarked characters from the training dataset to try and side-step getting sued by companies with enough resources to move the needle on what was allowed and disallowed in terms of digital art creation (since a ruling could disrupt their entire business model). I assume Emad Mostaque did the risk calculus differently when they open-sourced Stable Diffusion (I don't know for sure, but it smells like his attitude on the question was "I don't care because now that it's open-source nobody can delete it from the Internet anyway").

minimaxir 4y ago
That's true, although I'd still expect something a little blue. I had similar issues with VQGAN + CLIP, where CLIP is also trained by OpenAI but otherwise handles copyrighted characters a bit better.
- shadowgovt 4y ago
  If Sonic the Hedgehog was specifically excluded from the dataset, DALL-E 2 has no way to know the hedgehog should be blue. Instead, I think it's interpreting "a portrait of Sonic the Hedgehog" as a hedgehog that happens to have a name, and almost all hedgehogs, regardless of their name, are the same color. Even if someone had added a blue realistic-looking hedgehog to the dataset, if they referenced Sonic their image would have been deny-listed. So the only way to get there is to add the adjectives one is looking for, such as `a portrait of a blue hedgehog wearing red sneakers and eating a chili dog`... And, indeed, when I try that prompt, I definitely get some Sonic-alikes.
mg 4y ago
Dall-E 2 can render Sonic the Hedgehog just fine if you give it a bit more context: https://twitter.com/illubots/status/1571921642569502720 https://twitter.com/illubots/status/1571921642569502720

elteto 4y ago

Quick question: what is it with tagging these SD prompts with “trending on artstation”? Why is specifically artstation important?

beschizza 4y ago
A known dataset source with a strong quality filter
faldore 4y ago
and Greg Rutkowski too
- minimaxir 4y ago
  To add on this: I specifically avoided using him (or other living artists) in the prompts for this post since that's been a hot issue lately: https://www.technologyreview.com/2022/09/16/1059598/this-artist-is-dominating-ai-generated-art-and-hes-not-happy-about-it/ https://www.technologyreview.com/2022/09/16/1059598/this-art... I did test it though, which resulted in...this: https://twitter.com/minimaxir/status/1571520043737042946 https://twitter.com/minimaxir/status/1571520043737042946
PeterisP 4y ago
It turned out to be a simple way to communicate to the model "I want this to be similar to high quality images only, instead of all kinds of shitty art you've seen".

macrolime 4y ago

Gonna have to pick up one of those cheap RTX 3090s that miners are getting rid of so I can try this myself. With 8GB I'm just getting out of memory errors.

14 4y ago
I was hoping an online service would pop up where you can use someone else's hardware for a small fee and generate images. I really want to give it a try but only have a gtx970 so don't think it will work.
- kcartlidge 4y ago
  > I was hoping an online service would pop up where you can use someone else's hardware for a small fee and generate images. This one charges but gives pretty good results at 512x512 - and in only a couple of seconds at that resolution. For more logo/cartoony stuff you can generally get a 10 step done in less than a second. When I signed up a few weeks ago they gave 200 credits free (IIRC). After that it is $10 for another 1,000 credits (a credit gives 5 images at 10 step 512x512 or 1 image at 50 step 512x512). https://beta.dreamstudio.ai https://beta.dreamstudio.ai Only thing to note is that after a while it slows down, but I realised that is because every image generated is going into an array in local storage in the browser. Using the browser's Inspect -> Application area to clear that array every 100 images or so sorted that out.
- pbhjpbhj 4y ago
  Places remote computer providers like Vultr, Digital Ocean, Linode, etc., have servers with GPUs you can run up. You pay for uptime, and storage (including machines that are not up).
montebicyclelo 4y ago
You can 100% run this on an 8gb card. Make sure to load the model weights in half precision (and optionally enable attention slicing). E.g., using the diffusers library [1]: from diffusers import StableDiffusionPipeline from torch import autocast pipe = StableDiffusionPipeline.from_pretrained( "CompVis/stable-diffusion-v1-4", revision="fp16", torch_dtype=torch.float16, use_auth_token=True ) pipe = pipe.to("cuda") pipe.enable_attention_slicing() prompt = "a photo of an astronaut riding a horse on mars" with autocast("cuda"): image = pipe(prompt).images[0] [1] https://github.com/huggingface/diffusers https://github.com/huggingface/diffusers

sophrocyne 4y ago

Stable Diffusion is wild - the space has been quickly developing and watching the pace of development makes me reconsider what I consider "staggering". I've been blown away. The accessibility of this technology is even more incredible - there's even a fork that is working on M1 Macs (https://github.com/lstein/stable-diffusion https://github.com/lstein/stable-diffusion) We are in for some interesting times. Whatever the next iteration of Textual Inversion is will be extremely disruptive, especially if the concepts continue to be developed collectively.

14 4y ago

Is there an online service where I could use the features of Stable Diffusion to create images? I don't think my gtx970 meets the specs needed but I am a total noob and not sure. I have been dying to try it out however and would like to find a way to make a few images. If anyone knows of any services I would be thankful for a link. Thanks

kaeruct 4y ago
I like beta.dreamstudio.ai

wombat-man 4y ago

I kinda wish they'd release that version as a special feature. Probably not worth the effort though

dougmwne 4y ago

How many images do you need to get a good textual inversion? If you were to draw a custom character in a few poses, could you then generate infinite new styles and poses of that character? Or does it have to be something already common in the training dataset that you are just putting a very specific noun to?

Filligree 4y ago
Five or six is recommended.
- skinner_ 4y ago
  If someone played around with textual inversion: what happens if I give it hundreds of images? Does it plateau, does it regress, and if it regresses, what is the issue exactly?

cercatrova 4y ago

Anyone else getting bored of generating images? I used Stable Diffusion even before the public release (there was a guide on 4chan with leaked weights) and I'd been using it for a few weeks. I just feel like I've generated all the images I want to for now, and I just don't have any more interesting concepts I want to explore anymore. When you have so much variety in images, I got acclimatized to them and they all became the same and uninteresting after a while. It's like a hedonic treadmill but for AI image generation. I assume this will have the most usage as a once in a while tool for artists to get inspired from, and also a tool for commercial usage such as in a Photoshop or Figma plugin for designers. The lay person who wants to generate images will get bored after a while.

spywaregorilla 4y ago
No particular reason to think this is suddenly going to make you want to be an artist
illubots 4y ago
The first human who discovered that they could draw something on a wall probably also got bored after a while. And thought "I assume this will have the most usage as a once in a while tool to decorate my cave with funny animals". But then drawing animals turned into writing. Turned into printing. Turned into emails and the web and the web 2.0 and now here we are, sending our thoughts across the world with the speed of light. Using software that we wrote to tell computers how to do this for us. It might be similar with AI driven image generation. That it evolves in ways that are currently hard to foresee. Maybe one day, our thoughts are translated in realtime into images and send across the galaxy with the speed of light. Or faster?
- BizarroLand 4y ago
  How cool will it be when we can tell a program, "I want a fantasy story that continues on Lord of the Rings and details Pippin and Merry's second great and wonderful adventure after Frodo left middle earth" and get a 300 page story that is well-written and inventive and fun to read. I know the requirements for something like that are extraordinarily high, but I could see it in 30-40 years being viable.

bscphil 4y ago

> Indeed, there are many images of Sonic in the training dataset, however the generated images do not verbatim reproduce or otherwise plagiarize results from the training set above (I checked each one). This understanding of plagiarism reminds me of what students tell me when I ask them why they thought they could paraphrase an uncited Wikipedia article as their paper, right before I report them to the dean's office.

minimaxir 4y ago
Plagiarism in art is more complicated than plagiarism in a thesis paper, especially since art is a creative field defined by inspiration and iteration. I added that line because reproduction of the input dataset into an AI is a valid concern (such as reproducing a Getty Photos watermark), but that isn't happening here.

spywaregorilla 4y ago

That's an ugly sonic. It doesn't really look like "Ugly Sonic" though.

King-Aaron 4y ago

Just an aside, if you want to see Ugly Sonic resurrected, just watch the new Chip N Dale movie... It's actually an alright film, and the fact that they put ugly sonic in it was pretty amazing