8 ms·
Show HN: Generate Stable Diffusion scenes around 3D models
3D-to-photo is an open source tool for Generative AI product photography, that uses 3D models to allow fine camera angle control in generated images.
If you have 3D models created using the iOS 3D scanner you can upload them directly on to 3D-to-photo and describe the scene you want to create. For example:
"on a city side walk"
"near a lake, overlooking the water"
Then click "generate" to get the final images.
The tech stack behind 3D-to-photo:
Handling 3d models on the web: @threejs
Hosting the diffusion model: @replicate
3D scanning apps: shopify,Polycam3D or LumaLabsAI
- halyconWays 3y agoThis sounds pretty cool, do you have a demo or maybe a webm to put in the README.md?
- pj_mukh 3y agohaha a costly demo to run with the hn hug + inference costs..
- neilxm 3y agoyeaaa. well i could technically add a settings page to add your Replicate api token and run it on Vercel.
- neilxm 3y agoGood point, I haven't put up a web demo you can try without running the repo, but there's a video linked to the image on the very top of the README. here's the video again, i should probably make it more obvious that the image is clickable :) https://www.youtube.com/watch?v=P3yPn92v3u8 https://www.youtube.com/watch?v=P3yPn92v3u8
- chankstein38 3y agoHow is this different than just photoshopping a 2d image of a 3d object onto an SD generated background? Is it just meant to let people skip the step of generating a background and compositing? (Sorry, inpainting. But the distinction seems minimal here as people have been photoshopping 3d objects believably into scenes for decades before SD came around)
- brucethemoose2 3y agoThis streamlines the process of pasting a cropped render in, and Stable Diffusion frequently doesn't get the context right, so I assume thats useful for rapidly testing the shoe in different angles.
- neilxm 3y agoyea so as of now, it is a fairly loose integration between threejs "rendering" a 2d image of a scanned product at some orientation, and then Stable diffusion inpainting a background around it. However, there's a number of extensions here, that makes the integration of 2D and 3D more interesting going forward 1. 3D models means you can relight the model before running it through in-painting. adding lights in 3D around the product, plus using a more raycasted rendering system (which is now possible in-browser) means you can control the input to SD really well. 2. This one is the most interesting piece. You can create a 3D editor to allow very simple low poly style scene creation, lets say a pedestal and a vase or something from a product photography standpoint - then pass the depth map or canny edges as conditioning through something like control net and you have a super controllable scene design tool that you can finely control - both in camera angle and perspective.
- pj_mukh 3y agoAlso this is inpainting, not compositing.
- chankstein38 3y agoIs it really very different? Mind you just slapping the subject and a random background together would, accurately, not produce a very good result but for decades before SD existed people were able to accurately composite objects into images matching shadows and manipulating lighting to fit. The distinction doesn't feel worth commenting about overall.
- pj_mukh 3y agoInpainting manually is incredibly laborious. The comparison is almost like manually adding numbers on a paper spreadsheet vs excel. Manual inpainting gives a lot of control though, something I hope these SD pipelines can improve on.
- causi 3y agoGod we're so close from being able to feed a photo and some measurements into a program and get an accurate model out of it. I can't wait until my smartphone and a set of calipers can replace a $700 3d scanner.
- chankstein38 3y agoFor what it's worth, this doesn't look like it's anything towards that. This is just letting you manipulate a 3d model into an angle, save a 2d image of it, and put it into a generated background. This doesn't turn 2d->3d or seem to do anything 3d except allow a 3d model to be loaded for a photoshoot.
- neilxm 3y agoPretty much. additionally, i think NeRF is really promising for this use case. You don't really need a 3D model per se to view a thing from different angles. The only issue is lighting - both photogrammetry and nerf suck have baked lighting which really limits what you can do with the model. But there's a bunch of work towards solving that problem, so I can see that problem going away soon!
- yieldcrv 3y ago3D Guassian Splatting seems to be superseding NeRF NeRF might be a dead end technique in case your trying to keep up
- cvhashim04 3y agoWow this is insanely cool
- neilxm 3y agothank you!
- mlboss 3y agoWhy does it need a 3d model ? It looks like it is just doing inpainting which can be done with a single image.
- neilxm 3y agoYea you don't absolutely need a 3D model. The benefit of 3D though, is you can define any camera angle and perspective of the main product. now of course you just take a picture of a product from any angle if it were in front of you, but that's not always feasible. the use case here is scalable photo generation for ecomm stores with thousands of inventory items. Additionally, 3d means more than just camera angle control. you can define a scene in 3D and send it into control net to produce a very specific image
- mlboss 3y agoyeah that makes sense. I think the challenge is how much you can automate vs the quality.
- jcims 3y agoAutomatic lifelike shadows appears to be one benefit.
- neilxm 3y agoYea! Shadows on the item or shadows propogated back onto the scene
- astrange 3y agoIt doesn't appear that this project uses it, but a 3D model would give you all the other information like depth maps/normal maps that you would need to light the object itself properly. i.e. change the pixels of the object and not just draw a background around it.
- neilxm 3y agoThat’s the idea for next steps. Basically if you generate a backdrop and then estimate light direction you can inverse render that onto the 3d model given all the depth information you get for free from the model
- spiderxxxx 3y agohow is the lighting on the model? I assume you can't do anything other than overcast days, because the lighting isn't specified.
- neilxm 3y agoSo because this is a threejs canvas, we could build an interface to add lights anywhere, to adjust lighting on the model. this is an early version so it's not in there, but really easy to add in code. perhaps that will be the next feature.
- bsenftner 3y agoYou need to use your Stable Diffusion generated background environment to also generate spherical environment illumination maps so you can do image based lighting. That enables the 3D model(s) to correctly match any environment imaginable by projection mapping the environment onto the model; after which the model's materials define if that mapping is reflected, mixed with the underlying surface color, ignored, or some combination percentage. It's how integrated CGI is done in film VFX.
- yanma 3y agoNeed gaussian splatting integrated asap https://huggingface.co/blog/gaussian-splatting https://huggingface.co/blog/gaussian-splatting
- neilxm 3y agoHah. Yep!
- deleted 3y ago[deleted]
- chrisnight 3y agoGiven that Stable Diffusion is designed to be able to run on consumer hardware, without the need for a third party cloud platform, it saddens me to see that this, alongside many other similar projects, require the use of a third party platform for hosting the model, even for local usage. The tool itself does seem interesting though.
- neilxm 3y agoSo the api used for inpainting is very easily swappable to a local instance. But when you say consumer hardware, I think you might be overestimating how capable most people’s computing setups are.
- Folcon 3y agoIsn't that to a degree a function of time? There may be a point when that's not the case, but at present this stuff will run on hardware accessible to a consumer, assuming they're ok waiting a bit.
- neilxm 3y agoyea for sure. i guess i'd like this to be something usable on mobile as well. but directionally you're right. there's a lot to be gained from optimizing for local compute in many applications. something to look into.
- yieldcrv 3y agoI took a quick look at the python flask code and I’m still not sure if there’s a reason of not just using Next’s server side aspects. JS can do every operation I skimmed by Thoughts?
- neilxm 3y agoYes and no. So I did it because I honestly get annoyed with pixel operations in node js. The same thing in Python with the pillow library achieves the same things in quarter the lines of code as you would need in something like sharp in nodes The other reason I used a Python backend was that I want to extend it to more involved image processing, like producing control net inputs or post processing the end result. Does that make sense ? Open to better ideas though
- tinytera 3y agoLooks pretty cool. Can anyone comment on how to hack together the opposite? That is, going from 2D object image to 3D rendering with in-painted background? Or is that not possible right now.
- pj_mukh 3y ago2D to 3D networks are pretty amazing right now. High poly meshes and low poly textures are possible.
- quitit 3y agoDo you mean transforming a sketch to a 3D-looking image(i.e not a 3d mesh model): If so Stable diffusion with control net can do that using a good prompt and the SDXL model. Do you mean you have an existing photo of something and would like to add a realistic setting. There's a lot of ways to do this, but probably the easiest right now is the Generative Fill feature in Adobe Photoshop (beta).
- yanma 3y agoNot sure if they are planning on releasing this but you can mix a image2Nerf model (threestudio is a good repo for this) for the object 3d model, and an image2depth model like ZoeDepth to generate 3d background.
- product-render 3y ago[dead]
- scotty79 3y agoNeat. I recently was having fun with doing something similar manually in Blender, generating depth map and using it in Stable Diffusion with controlnet. Results were great. My models didn't have texture though, so sd generated it. But I imagine I could go with img2img to preserve texture if I had it.
- miachelbrown 3y ago[dead]