7 ms·

Stable Diffusion Text-Prompt-Based Inpainting – Replace Hair, Fashion

136 points by amrrs 4y ago

londons_explore 4y ago
Why does the mask need to be binary? Surely it's possible to have a full alpha mask, such that 50% alpha means "push the diffusion process towards this value, but don't force it to generate this value".
- phire 4y ago
  Surely you don't need to? Just over-expand the mask and let stable diffusion decide what it's going to keep and what it's going to replace.
  - XorNot 4y ago
    You'd effectively be trying to do a pixel-by-pixel diffusion strength parameter, which I'm not sure has a coherent interpretation to the algorithm (because it's currently a scalar applied to the run settings).
- bryced 4y ago
  the "alpha" is already built into the tooling and specified independent of the mask. IE. the stable diffusion inpainting takes hints from what you leave and "decides" what to keep
fariszr 4y ago
The progress in the AI space is absolutely astounding. In less then a year, we went from no AI photo generation(from prompts), to DALL-E2 a commercial service, then competitors started popping up like mid journey, and now we have Stable diffusion, which is a source available AI you can run your self, unlocking implementations like this. There are other companies now hyping AI video generation like runaway(1) https://twitter.com/runwayml/status/1568220303808991232 https://twitter.com/runwayml/status/1568220303808991232
- d3ckard 4y ago
  Totally disagree. A whole of AI business space seems totally focused on pushing the boundaries of what is possible, completely ignoring delivering something consistently useful. I played a bit with image generation recently and most results were abysmal. Sure, it can create great things and prompt hacking will be a thing for a while. It’s however very far from “for each prompt I get a working (as in not broken with artifacts) and matching image”. IMO business usability depends on average case mostly and this hasn’t impressed me yet. The elephant in the room is the “black box” nature of all neural networks. They are not interpretable for humans, therefore we cannot know when they can royally screw up. That means unless we keep a human in the loop, it’s hard to really integrate it into anything critical. And keeping humans out of the loop is what most AI companies promised as an end goal. Basically, I am embracing incoming AI winter. Not because no great progress has been made, but because what was promised will never be delivered (as have been the case previous times). At the same time, AI is here to stay and AI based tools are going to become common place. It will just be less of a great deal than everybody expected.
  - joyfylbanana 4y ago
    It is how you measure "progress". For you progress seems to be only about business. I think majority of people (including me) are just delighted about this new toy that brings them joy. In my life it is great progress if I get new innovative toys that haven't been available before.
    - d3ckard 4y ago
      How did you get to conclusion I think progress is mostly about business? My point is about promises that funded current wave of AI craziness do not seem to get fulfilled. At the precise moment it becomes obvious for everybody, funding will stop and bet on the next horse, whatever it will be. It seems that autonomous driving got stuck on “driver must be ready to take control”. If that doesn’t change, Uber is just a glorified tax corporation. Tesla car revolution ain’t happening, if my car can’t drive me to a spot without assistance (allowing me to sleep or be drunk or whatever). They become just another car company with a head start on electric. And the rest of the AI industry seems to follow the pattern for me - great results (cars can mostly drive themselves nowadays, that’s insane!), but always a notch less than expected. Because what was expected were humans replacements and what we got is human augmenters. It’s probably better for humanity, as productivity will rise, but humans won’t be cut from the loop. I just don’t think it is this particular result that Big Money had in mind when they poured money over it.
bryced 4y ago
This python library does the same thing (but didn't get traction when I posted it yesterday): https://github.com/brycedrennan/imaginAIry#automated-replacement-txt2mask-by-clipseg https://github.com/brycedrennan/imaginAIry#automated-replace... https://news.ycombinator.com/item?id=32887385 https://news.ycombinator.com/item?id=32887385 And I got the idea from here: https://github.com/ThereforeGames/txt2mask https://github.com/ThereforeGames/txt2mask Which is using the model here: https://github.com/timojl/clipseg https://github.com/timojl/clipseg Clipseg is doing the hard part!
- stavros 4y ago
  This looks great! I've been looking for a Python library to use with Phantasmagoria[1] for ages, but everyone is doing web UIs. You even packaged it up in a Docker container, very nice, thank you! [1]: https://phantasmagoria.stavros.io https://phantasmagoria.stavros.io
- amrrs 4y ago
  This is so cool. Everything in one command, Thanks for sharing. How are you serving the model? I couldn't see it on the Github repo.
  - bryced 4y ago
    For code that doesn't have good pypi packaging I vendor it using scripts in the makefile
- em_ 4y ago
  Thanks for the link. I created a quick Colab Notebook for those without sufficient hardware: https://colab.research.google.com/drive/1J4yeGLk2MRXrD5lOoS546ehxd0m_fRKE?usp=sharing https://colab.research.google.com/drive/1J4yeGLk2MRXrD5lOoS5...
  - bryced 4y ago
    oh that's great - thank you!
seydor 4y ago
> Enhance 34 to 36. Pan right and pull back. Stop. Enhance 34 to 46. Pull back. Wait a minute, go right, stop. Enhance 57 to 19. Track 45 left. Stop. Enhance 15 to 23. Give me a hard copy right there.
MikeYasnev007 4y ago
Sharing masks and code open sourcing is a arguable question. I definitely don't share anything outside of commercial company
MikeYasnev007 4y ago

136 points by amrrs 4y ago

Why does the mask need to be binary? Surely it's possible to have a full alpha mask, such that 50% alpha means "push the diffusion process towards this value, but don't force it to generate this value".

phire 4y ago
Surely you don't need to? Just over-expand the mask and let stable diffusion decide what it's going to keep and what it's going to replace.
- XorNot 4y ago
  You'd effectively be trying to do a pixel-by-pixel diffusion strength parameter, which I'm not sure has a coherent interpretation to the algorithm (because it's currently a scalar applied to the run settings).
bryced 4y ago
the "alpha" is already built into the tooling and specified independent of the mask. IE. the stable diffusion inpainting takes hints from what you leave and "decides" what to keep

fariszr 4y ago

The progress in the AI space is absolutely astounding. In less then a year, we went from no AI photo generation(from prompts), to DALL-E2 a commercial service, then competitors started popping up like mid journey, and now we have Stable diffusion, which is a source available AI you can run your self, unlocking implementations like this. There are other companies now hyping AI video generation like runaway(1) https://twitter.com/runwayml/status/1568220303808991232 https://twitter.com/runwayml/status/1568220303808991232

d3ckard 4y ago
Totally disagree. A whole of AI business space seems totally focused on pushing the boundaries of what is possible, completely ignoring delivering something consistently useful. I played a bit with image generation recently and most results were abysmal. Sure, it can create great things and prompt hacking will be a thing for a while. It’s however very far from “for each prompt I get a working (as in not broken with artifacts) and matching image”. IMO business usability depends on average case mostly and this hasn’t impressed me yet. The elephant in the room is the “black box” nature of all neural networks. They are not interpretable for humans, therefore we cannot know when they can royally screw up. That means unless we keep a human in the loop, it’s hard to really integrate it into anything critical. And keeping humans out of the loop is what most AI companies promised as an end goal. Basically, I am embracing incoming AI winter. Not because no great progress has been made, but because what was promised will never be delivered (as have been the case previous times). At the same time, AI is here to stay and AI based tools are going to become common place. It will just be less of a great deal than everybody expected.
- joyfylbanana 4y ago
  It is how you measure "progress". For you progress seems to be only about business. I think majority of people (including me) are just delighted about this new toy that brings them joy. In my life it is great progress if I get new innovative toys that haven't been available before.
  - d3ckard 4y ago
    How did you get to conclusion I think progress is mostly about business? My point is about promises that funded current wave of AI craziness do not seem to get fulfilled. At the precise moment it becomes obvious for everybody, funding will stop and bet on the next horse, whatever it will be. It seems that autonomous driving got stuck on “driver must be ready to take control”. If that doesn’t change, Uber is just a glorified tax corporation. Tesla car revolution ain’t happening, if my car can’t drive me to a spot without assistance (allowing me to sleep or be drunk or whatever). They become just another car company with a head start on electric. And the rest of the AI industry seems to follow the pattern for me - great results (cars can mostly drive themselves nowadays, that’s insane!), but always a notch less than expected. Because what was expected were humans replacements and what we got is human augmenters. It’s probably better for humanity, as productivity will rise, but humans won’t be cut from the loop. I just don’t think it is this particular result that Big Money had in mind when they poured money over it.

bryced 4y ago

This python library does the same thing (but didn't get traction when I posted it yesterday): https://github.com/brycedrennan/imaginAIry#automated-replacement-txt2mask-by-clipseg https://github.com/brycedrennan/imaginAIry#automated-replace... https://news.ycombinator.com/item?id=32887385 https://news.ycombinator.com/item?id=32887385 And I got the idea from here: https://github.com/ThereforeGames/txt2mask https://github.com/ThereforeGames/txt2mask Which is using the model here: https://github.com/timojl/clipseg https://github.com/timojl/clipseg Clipseg is doing the hard part!

stavros 4y ago
This looks great! I've been looking for a Python library to use with Phantasmagoria[1] for ages, but everyone is doing web UIs. You even packaged it up in a Docker container, very nice, thank you! [1]: https://phantasmagoria.stavros.io https://phantasmagoria.stavros.io
amrrs 4y ago
This is so cool. Everything in one command, Thanks for sharing. How are you serving the model? I couldn't see it on the Github repo.
- bryced 4y ago
  For code that doesn't have good pypi packaging I vendor it using scripts in the makefile
em_ 4y ago
Thanks for the link. I created a quick Colab Notebook for those without sufficient hardware: https://colab.research.google.com/drive/1J4yeGLk2MRXrD5lOoS546ehxd0m_fRKE?usp=sharing https://colab.research.google.com/drive/1J4yeGLk2MRXrD5lOoS5...
- bryced 4y ago
  oh that's great - thank you!

seydor 4y ago

> Enhance 34 to 36. Pan right and pull back. Stop. Enhance 34 to 46. Pull back. Wait a minute, go right, stop. Enhance 57 to 19. Track 45 left. Stop. Enhance 15 to 23. Give me a hard copy right there.

MikeYasnev007 4y ago

Sharing masks and code open sourcing is a arguable question. I definitely don't share anything outside of commercial company

MikeYasnev007 4y ago