7 ms·
Stable Diffusion 2 Depth Guided model: architecture photos from dollhouse
- liuliu 4y agodepth2img is underrated for SDv2 release, probably because AUTOMATIC1111 integration is done only 1 or 2 weeks ago. Later this week, Draw Things will have a release with depth2img model as well, initially only supporting iOS-provided depth map, but will do MiDaS inferred depth map soon. Going to be exciting to see what people comes up with.
- Der_Einzige 4y agoDoes this work with dreambooth automatic webUI extension yet? 2.1 doesn't work
- deleted 4y ago[deleted]
- liuliu 4y agoI think, we may be able to apply LoRA trick to transfer custom dreambooth models (trained with SD v2) back to depth guide models.
- DethNinja 4y agoDepth Guided models are amazing for game art texturing: https://hackaday.com/2022/12/18/image-generating-ai-can-texture-an-entire-3d-scene-in-blender/ https://hackaday.com/2022/12/18/image-generating-ai-can-text... I already incorporated this to Houdini to generate 3D distance meshes and it works wonders. If anyone is interested in experimenting with this, I really recommend Hugging Face diffusers library. I wonder when we will be able to generate studio quality 3D models from a single 2D image. I know there are solutions that can do this but it is still nowhere near professional quality.
- robertlagrant 4y agoIt looks like the artwork from that game Psychonauts. Awesome.
- corysama 4y agoReally obvious use that took too long to occur to me: Make a bunch of textures for your game. Label them and use them to train a specialized model file. Use that to make more textures for your game. It’ll get you 80% of the way there for each texture really fast. Would work really well if you are making a ton of NPCs and made a head model and a separate clothes model.
- dwallin 4y agoThis is really cool, and I can immediately see a lot of way to take this further and improve some of it's weaknesses. To solve the issue with hidden faces (right now it just projects out copies of the visible textures, you could combine this with inpainting: Rotate the scene / model to expose untextured faces sides that get masked out and inpainted, then rinse and repeat until the model is fully textured. Another interesting (albeit much harder) thing to try would be to take the image output from SD, run it through MiDaS to get a new depth map with additional details. Diff the depth maps in 3d space and update the geometry to match before projecting the image. Combine it with the first suggestion and you would have a process for progressively refining a detailed, fully textured model.
- bryced 4y ago> Another interesting (albeit much harder) thing to try would be to take the image output from SD, run it through MiDaS to get a new depth map with additional details. Diff the depth maps in 3d space and update the geometry to match before projecting the image. Combine it with the first suggestion and you would have a process for progressively refining a detailed, fully textured model. I made a go at this exact approach last night. I'm new to working with 3d data but at least got a mesh rendered from the depth map. Spent most of my time fighting with tooling. Let me know if you want to help out.
- 4y ago
- joewhatkins 4y agoWe got this working in Three.js - it works great for painting scenery and backgrounds.
- FarhadG 4y agoCare to share more? Would love to see how it's being leveraged in that space
- matthewfcarlson 4y agoI'd love to see this as well
- isoprophlex 4y agoThe pace and extent of innovation in this space is breathtaking. Almost every day there's a new super original idea showing up, that builds on something we just simply didn't have four months ago. The open sourcing of these models was of course instrumental to this. So thanks Huggingface et al., I guess!
- charcircuit 4y agoStable Diffusion is not an open source model.
- bandana 4y agoIt is
- IanCal 4y agoIt's very clear how the discussion goes from here so it's really worth short circuiting it and providing the detail. The models are openly available but the license places some restrictions on the use. > You agree not to use the Model or Derivatives of the Model: > - In any way that violates any applicable national, federal, state, local or international law or regulation; > - For the purpose of exploiting, harming or attempting to exploit or harm minors in any way; > - To generate or disseminate verifiably false information and/or content with the purpose of harming others; > - To generate or disseminate personal identifiable information that can be used to harm an individual; > - To defame, disparage or otherwise harass others; > - For fully automated decision making that adversely impacts an individual’s legal rights or otherwise creates or modifies a binding, enforceable obligation; > - For any use intended to or which has the effect of discriminating against or harming individuals or groups based on online or offline social behavior or known or predicted personal or personality characteristics; > - To exploit any of the vulnerabilities of a specific group of persons based on their age, social, physical or mental characteristics, in order to materially distort the behavior of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm; > - For any use intended to or which has the effect of discriminating against individuals or groups based on legally protected characteristics or categories; > - To provide medical advice and medical results interpretation; > - To generate or disseminate information for the purpose to be used for administration of justice, law enforcement, immigration or asylum processes, such as predicting an individual will commit fraud/crime commitment (e.g. by text profiling, drawing causal relationships between assertions made in documents, indiscriminate and arbitrarily-targeted use). https://github.com/CompVis/stable-diffusion/blob/main/LICENSE https://github.com/CompVis/stable-diffusion/blob/main/LICENS... For many, this distinction is more of either an academic one or one where we are OK with those kind of restrictions. Where I live if I did many of these things I would be either criminally or civilly (the right phrasing?) liable so having a license that tells me I can't break the law is a little redundant. I think 4 is possibly legal but an edge case and the last one is nation state level. You are probably technically correct, but the context here is very important IMO.
- 101010101010101 4y ago
- prvc 4y agoI wish Stable Diffusion had focused on more fine art and professional photography in their training corpus, like DALL-E did in its own one. All outputs I've seen have a decidedly low-quality or amateurish look, which must have been picked up from somewhere.
- orbital-decay 4y agoThing is, the primary power of large models is not in the styles they memorized, but in the ability to be fine-tuned and guided by reference content. The focus is your choice, you can shift the model bias with any available finetuning method and a certain amount of reference images. 99% of the training has already been done for you, so all you need is a relatively subtle push achievable on a single GPU (or a few GPUs). Take a look at Analog Diffusion outputs, for example.
- dwringer 4y agoWith patience the guidance can even be done simply, without training, with enough prompts, blending of prompts, input images to img2img, etc. I'm not an expert on art history, but those who are seem able to get just about any style on any image with just the base stable diffusion model. Adding specific photography terms can emulate some of the results from the analog diffusion model in regular SD as well.
- prvc 4y agoIt has nothing to do with "style". DALL-E 2 consistently astonishes with its composition, framing, variety of visual elements, and so on, which tend to also be good in oil paintings and professional photographs, but not at all in the sort of content which predominates on art sharing apps with user-generated content.
- orbital-decay 4y agoIf you mean porn (which is censored in DALL-E), SD also can't make it as it's mostly filtered out from the training set. All of it is made by custom finetuned models, which is my point. Same with hands, which SD can struggle with. Styles are not the only thing that you can transfer, but subjects and concepts as well.
- andrewon 4y agoA primer to Depth model in Stable Diffusion 2: https://stable-diffusion-art.com/depth-to-image/ https://stable-diffusion-art.com/depth-to-image/
- monkeydust 4y agoCan this be used for virtual 2d try on of clothing?
- bzzzt 4y agoLooked impressive until I saw the guitars ;)
- system2 4y ago[flagged]
- codetrotter 4y agoIt’s ok. If you want you can read it in Threadreader instead. https://threadreaderapp.com/thread/1605276764598665217.html https://threadreaderapp.com/thread/1605276764598665217.html I personally prefer to just read it on Twitter directly these days if someone links to Twitter, as long as Twitter doesn’t pop up the thing asking me to sign in which it sometimes does. But I do secretly wish that everyone would migrate away from Twitter altogether and use Mastodon etc instead.
- folopf 4y agoMastodon is a cult though. I can't get over how irritatingly homogenous in opinion it is despite being federated, thanks to the draconian moderation policy that the large instances impose.
- mrd3v0 4y agoThere are hundreds if not thousands of instances which federate with each other. The entire point of Mastodon is to avoid giving too much power to one instance. Join a smaller one.
- xwdv 4y agoWhat if the smaller one isn’t accepted to the federation?
- kelseyfrog 4y agoHave more pro-social or socially acceptable behavior?
- pea 4y agoOn a similar note, has anyone had any luck using these models to try different paint colourings in their house (from uploading photos of their rooms)? I've struggled to do this with SD, and the non-AI apps aren't great at it either.
- herendin2 4y agoWhen you afford SD img2img influence with a high strength parameter, so it can change the colors of a photo, you also are letting it have enough influence to redesign the room at random and not just changing the colors Dreambooth trained on the ground truth photos might help a bit
- Auracle 4y agoI’d love for there to be a form of frequency separation, where we could keep the details but mess with the colors/less major details. Photographers have been attempting to emulate various films in different ways forever. Stable diffusion could do it absolutely perfectly.
- CuriouslyC 4y agoTry converting the images to grayscale, then setting the denoising strength very low and prompting for the exact colors you want things to be.
- nickthegreek 4y agoThis seems much easier in an app like photoshop/affinity if you want to keep everything really static. Or by using a combination of photoshop and then SD to clean up.
- usrusr 4y agoTake the depth field from life size furniture instead, all covered in chroma key surface and make "we never use the same crazy virtual studio twice, not even the arrangement" the identity of your fixed camera show?
- ramoz 4y agoIf you like using Colab, here’s a link to a notebook that spins up a nice simple demo of the depth model. https://github.com/backnotprop/Colab-Stable-Diffusion-2-Depth-UI https://github.com/backnotprop/Colab-Stable-Diffusion-2-Dept...
- itronitron 4y agocoming soon to an AirBnB listing near you :)
- chx 4y agoIsn't all AI art stolen art?
- xwdv 4y agoWhen will AI artists get fast enough that they can generate imagery as you type?