4 ms·

How is this different than just photoshopping a 2d image of a 3d object onto an SD generated background? Is it just meant to let people skip the step of generat

How is this different than just photoshopping a 2d image of a 3d object onto an SD generated background? Is it just meant to let people skip the step of generating a background and compositing? (Sorry, inpainting. But the distinction seems minimal here as people have been photoshopping 3d objects believably into scenes for decades before SD came around)

brucethemoose2 3y ago
This streamlines the process of pasting a cropped render in, and Stable Diffusion frequently doesn't get the context right, so I assume thats useful for rapidly testing the shoe in different angles.
- neilxm 3y ago
  yea so as of now, it is a fairly loose integration between threejs "rendering" a 2d image of a scanned product at some orientation, and then Stable diffusion inpainting a background around it. However, there's a number of extensions here, that makes the integration of 2D and 3D more interesting going forward 1. 3D models means you can relight the model before running it through in-painting. adding lights in 3D around the product, plus using a more raycasted rendering system (which is now possible in-browser) means you can control the input to SD really well. 2. This one is the most interesting piece. You can create a 3D editor to allow very simple low poly style scene creation, lets say a pedestal and a vase or something from a product photography standpoint - then pass the depth map or canny edges as conditioning through something like control net and you have a super controllable scene design tool that you can finely control - both in camera angle and perspective.
pj_mukh 3y ago
Also this is inpainting, not compositing.
- chankstein38 3y ago
  Is it really very different? Mind you just slapping the subject and a random background together would, accurately, not produce a very good result but for decades before SD existed people were able to accurately composite objects into images matching shadows and manipulating lighting to fit. The distinction doesn't feel worth commenting about overall.
  - pj_mukh 3y ago
    Inpainting manually is incredibly laborious. The comparison is almost like manually adding numbers on a paper spreadsheet vs excel. Manual inpainting gives a lot of control though, something I hope these SD pipelines can improve on.
    - chankstein38 3y ago
      That's why I had asked if the goal of the project was just to skip the laborious parts of compositing! haha I'm with you on the lack of control of SD. It can create some amazing stuff but so frequently it's kind of just "put in some words and click until you get something you like then pull into an image editor to fix up issues" which ends up meaning I have to do some of the labor still anyway.
  - bogwog 3y ago
    > The distinction doesn't feel worth commenting about overall. That's like saying that the distinction between the output of stable diffusion and a real artist isn't worth commenting about overall, since they're both just paintings.
    - chankstein38 3y ago
      In the context of whether or not a process is compositing or inpainting, that's factual. The output is a 2d plate placed in front of a background. Both achieve similar results. The level of manual (or automated) effort is not what's being discussed here. So I'm not sure what point you're trying to make.
neilxm 3y ago
Reading your question again, I should clarify - it's not quite compositing a random background under the 2d image. it's using stable diffusion to "in-paint" a background that makes sense spatially, with correct shadows, lighting and perspective to match your prompt. That being said, there's still a lot to be done to deeply integrate the 3D and 2D spaces and that's just going to be part of the ongoing exploration in this project.
- chankstein38 3y ago
  That's fair. I guess in my head they're still one in the same. This is compositing without the step of having to match shadows and find a background with a decent angle, etc. I had asked in my original post if this was to skip the step of compositing (because you would do things like fix shadows etc if you were compositing) and that's what it sounds like it is, which is cool. Do you communicate anything about the angle or anything to SD? (outside of just giving it the image with a transparent background) I guess, given the additional context (and vs discussing the semantics of compositing), the better question is, how does this extend the capabilities of stable diffusion inpainting? Is it any different than just putting your 3d model into photoshop or in a 3d viewer, exporting with a transparent background, and inpainting around it?
  - neilxm 3y ago
    It's not that different right now you're right about that. It chains that sequence of steps together and I'm not really sending any meta data about the objects pose to SD, I just leave it up to the model. But the next step here is leveraging more control-net approaches and thats when it starts to really take advantage of 3D more than just the 2D snapshot minus background. For example, a 3D editor to allow for simple low poly style scene creation, that then serves as conditioning input to control net. For example staging a model of a chair in a sketchup model of a living room with super basic furniture elements in low poly 3D models. You pass that into SD and out comes a fully rendered image. At that point i think you could argue that stable diffusion could be used as a platform agnostic renderer like VRAY but for any 3D modelling tool. This was my version 1 haha
scotty79 3y ago
You can adjust the pose of a thing you are pasting.

brucethemoose2 3y ago

This streamlines the process of pasting a cropped render in, and Stable Diffusion frequently doesn't get the context right, so I assume thats useful for rapidly testing the shoe in different angles.

neilxm 3y ago
yea so as of now, it is a fairly loose integration between threejs "rendering" a 2d image of a scanned product at some orientation, and then Stable diffusion inpainting a background around it. However, there's a number of extensions here, that makes the integration of 2D and 3D more interesting going forward 1. 3D models means you can relight the model before running it through in-painting. adding lights in 3D around the product, plus using a more raycasted rendering system (which is now possible in-browser) means you can control the input to SD really well. 2. This one is the most interesting piece. You can create a 3D editor to allow very simple low poly style scene creation, lets say a pedestal and a vase or something from a product photography standpoint - then pass the depth map or canny edges as conditioning through something like control net and you have a super controllable scene design tool that you can finely control - both in camera angle and perspective.

pj_mukh 3y ago

Also this is inpainting, not compositing.

chankstein38 3y ago
Is it really very different? Mind you just slapping the subject and a random background together would, accurately, not produce a very good result but for decades before SD existed people were able to accurately composite objects into images matching shadows and manipulating lighting to fit. The distinction doesn't feel worth commenting about overall.
- pj_mukh 3y ago
  Inpainting manually is incredibly laborious. The comparison is almost like manually adding numbers on a paper spreadsheet vs excel. Manual inpainting gives a lot of control though, something I hope these SD pipelines can improve on.
  - chankstein38 3y ago
    That's why I had asked if the goal of the project was just to skip the laborious parts of compositing! haha I'm with you on the lack of control of SD. It can create some amazing stuff but so frequently it's kind of just "put in some words and click until you get something you like then pull into an image editor to fix up issues" which ends up meaning I have to do some of the labor still anyway.
- bogwog 3y ago
  > The distinction doesn't feel worth commenting about overall. That's like saying that the distinction between the output of stable diffusion and a real artist isn't worth commenting about overall, since they're both just paintings.
  - chankstein38 3y ago
    In the context of whether or not a process is compositing or inpainting, that's factual. The output is a 2d plate placed in front of a background. Both achieve similar results. The level of manual (or automated) effort is not what's being discussed here. So I'm not sure what point you're trying to make.

neilxm 3y ago

Reading your question again, I should clarify - it's not quite compositing a random background under the 2d image. it's using stable diffusion to "in-paint" a background that makes sense spatially, with correct shadows, lighting and perspective to match your prompt. That being said, there's still a lot to be done to deeply integrate the 3D and 2D spaces and that's just going to be part of the ongoing exploration in this project.

chankstein38 3y ago
That's fair. I guess in my head they're still one in the same. This is compositing without the step of having to match shadows and find a background with a decent angle, etc. I had asked in my original post if this was to skip the step of compositing (because you would do things like fix shadows etc if you were compositing) and that's what it sounds like it is, which is cool. Do you communicate anything about the angle or anything to SD? (outside of just giving it the image with a transparent background) I guess, given the additional context (and vs discussing the semantics of compositing), the better question is, how does this extend the capabilities of stable diffusion inpainting? Is it any different than just putting your 3d model into photoshop or in a 3d viewer, exporting with a transparent background, and inpainting around it?
- neilxm 3y ago
  It's not that different right now you're right about that. It chains that sequence of steps together and I'm not really sending any meta data about the objects pose to SD, I just leave it up to the model. But the next step here is leveraging more control-net approaches and thats when it starts to really take advantage of 3D more than just the 2D snapshot minus background. For example, a 3D editor to allow for simple low poly style scene creation, that then serves as conditioning input to control net. For example staging a model of a chair in a sketchup model of a living room with super basic furniture elements in low poly 3D models. You pass that into SD and out comes a fully rendered image. At that point i think you could argue that stable diffusion could be used as a platform agnostic renderer like VRAY but for any 3D modelling tool. This was my version 1 haha

scotty79 3y ago

You can adjust the pose of a thing you are pasting.