14 ms·
A working implementation of text-to-3D DreamFusion, powered by Stable Diffusion
- jrmann100 4y ago[Edited] The original Dreamfusion project was discussed here a few days ago: https://news.ycombinator.com/item?id=33025446 https://news.ycombinator.com/item?id=33025446
- nobbis 4y agoTook a week, as predicted: https://twitter.com/DvashElad/status/1575614411834011651 https://twitter.com/DvashElad/status/1575614411834011651
- nobbis 4y agoKey step in generating 3D – ask Stable Diffusion to score views from different angles: for d in ['front', 'side', 'back', 'side', 'overhead', 'bottom']: text = f"{ref_text}, {d} view" https://github.com/ashawkey/stable-dreamfusion/blob/0cb8c0e0fdf8944cbb431983f08deb082a2e3f56/nerf/utils.py#L221 https://github.com/ashawkey/stable-dreamfusion/blob/0cb8c0e0...
- shadowgovt 4y agoI'm modestly surprised that those few angles give us enough data to build out a full 3D render, but I guess I shouldn't be too surprised, as that's tech that has had high demand and been understood for years (those kind of front-cut / side-cut images are what 3D artists use to do their initial prototypes of objects if they're working from real-life models).
- mhuffman 4y agoI don't think that NeRFs require too many image to make impressive results.
- deleted 4y ago[deleted]
- nobbis 4y agoDreamFusion doesn't directly build a 3D model from those generated images. It starts with a completely random 3D voxel model, renders it from 6 different angles, then asks Stable Diffusion how plausible an image of "X, side view" it is. It then sprinkles some noise on the rendering, makes Stable Diffusion improve it a little, then adjusts the voxels to produce that image (using differentiable rendering.) Rinse and repeat for hours.
- shadowgovt 4y agoThank you for the clarification; I hadn't grokked the algorithm yet. That's interesting for a couple of reasons. I can see why that works. It also implies that for closed objects, the voxel data on the interior (where no images can see it) will be complete noise, as there's no signal to pick any color or lack of a voxel.
- nobbis 4y agoYes, although not complete noise – probably empty. Haven't checked but assume there's regularization of the NeRF parameters.
- FeepingCreature 4y ago
- dwallin 4y agoGiven the way the language model works these words could have multiple meanings. I wonder if training a form of textual inversion to more directly represent these concepts might improve the results. You could even try teaching it to represent more fine grained degree adjustments.
- bergenty 4y agoI can’t wait to generate novel 3D models to CNC/3D print! Can these be exported out as STL/OBJs?
- moron4hire 4y agoIn the usage notes, there's a line that mentions # test (exporting 360 video, and an obj mesh with png texture) python main_nerf.py --text "a hamburger" --workspace trial -O --test So I guess so. That's pretty awesome.
- samstave 4y agoHeckin A! I just got my 3D printer and was a bit too tipsy to assemble it the day it arrived - and have several things I want to print… It will be interesting to experiment with describing the thing I want to print with text instead of designing it in SolidEdge and see what AI thinks…. I wonder if you can feed it specific dimensions? “A holder for a power supply for an e bike with two mounting holes 120mm apart with a carry capacity that is 5 inches long and 1.5 inches deep”
- wongarsu 4y agoWell, here's 16 attempts of regular Stable Diffusion with that prompt [1], and here's what it things a technical drawing of it might look like [2]. Maybe two papers down the line :D For now you might have more luck with something less specific. 1: https://i.imgur.com/RPNCwyM.png https://i.imgur.com/RPNCwyM.png 2: https://i.imgur.com/c9pfM8U.png https://i.imgur.com/c9pfM8U.png
- samstave 4y agoStill dope… but are those also obj or stl? I like my DALLE expressions of “masterchief as ventruvian man as drawn by da Vinci” And my “technical exploded diagrams of cybernetic eco skeleton suits in blueprint” Try those out?
- wongarsu 4y agoThere's a small gallery of success and failure cases here [1]. It certainly doesn't look as good as the original, yet. I wonder if that's due to the implementation differences noted, less cherry picking in what they show, or inherent differences between Imagen and Stable Diffusion. Maybe Imagen just has a much better grasp of how images translate to actual objects, where Stable Diffusion is stuck more on the 2d image plane. 1: https://github.com/ashawkey/stable-dreamfusion/issues/1 https://github.com/ashawkey/stable-dreamfusion/issues/1
- nopenopenopeno 4y agoOne cannot help but notice the success cases are expected to have symmetry along at least one axis, whereas the failure cases are not.
- namarie 4y agoAren't squirrels and frogs expected to have an axis of symmetry? I think the reason for the failures is the presence of faces; it seems to be trying to make a face visible from all angles.
- gpm 4y agoWhich probably has a lot to do with us taking nearly all our pictures of things with faces from the face facing direction.
- shadowgovt 4y agoI feel like the Cthulhu head is extra-successful, given the subject matter. Non-Euclidean back-polygon imaging? Good work, algorithm. ;)
- codeflo 4y agoI guess a lot can be done to force the model to create properly connected 3D shapes instead of these thin protruding 2D slices. But I noticed something else. Some of the angles “in between” the frog faces have three eyes. I wonder if part of the issue might be that those don’t look especially wrong to Stable Diffusion. It’s often surprisingly confused about the number of limbs it should generate.
- Cook4986 4y agoCurrently working with a student group to build out a 3D scene generator (https://github.com/Cook4986/Longhand https://github.com/Cook4986/Longhand), and the prospect of arbitrary, hyper-specific mesh arrays on demand is thrilling. Right now, we are relying on the Sketchfab API to populate our (Blender) scenes, which is an imperfect lens through which to visualize the contents of texts that our non-technical "clientele" are studying. Since we are publishing these scenes via WebXR (Hubs), we have specific criteria related to poly counts (latency, bandwidth, etc) and usability. Regarding the latter concern, it's not clear that our end users will want to wait/pay for compute. *copyedited
- sirianth 4y agowow
- GistNoesis 4y ago>Stable-Diffusion is a latent diffusion model, which diffuses in a latent space instead of the original image space. Therefore, we need the loss to propagate back from the VAE's encoder There is also an alternative way to handle this latent difference with the original paper that should also work : Instead of working in voxel color space, you push the latent to the voxel (Aka instead of having a voxel grid of 3d rgb color, you have a voxel grid of dimlatent latents, (you can also use spherical harmonics if you want as it works just the same in nd) ). Only the color prediction network differ, the density is kept the same. The NERF then directly render to the latent space (so there are less rays to render) which mean you need to decode it with the VAE only for visualization purposes and not in the training loop.
- hwers 4y agoThis sounds really interesting but I’m not sure I follow. Having a hard time expressing how I’m confused though (maybe its unfamiliar nerf terminology) but if you have the time I’d be very interested if you could reformulate this alternative method somehow (I’ve been stuck on this very issue for two days now trying to implement this myself).
- taneq 4y agoNERF is neural radiance fields (the neural 3D reconstruction method that nVidia published recently.) Basically if I’m reading it right, this does the synthesis in latent space (which describes the scene rather than rendering vocals) then translates it into a NERF. It sounds kind like the Stable Diffusion description that was on here earlier.
- going_ham 4y agoDo you have refereneces to NeRF papers that can directly compute the field in the latent code? Because it NeRF based methods are essentially performing solving the rendering equation to learn the mapping, what could be alternative equation for directly learning latent code? Your idea is interesting, could you elaborate on it!
- 4y ago
- sdwvit 4y agoAll these news on image/3d/video generation just show how we live in the middle of ai/ml breakthrough. Incredible to see news with extreme progress in the field like this popping everyday.
- dmingod666 4y agoBrowse Lexica.art your mind will be blown by the range and amount of detail on some of the art. Like this (nsfw content): https://lexica.art/?q=Intricate+goddess https://lexica.art/?q=Intricate+goddess There is an addictive and trippy quality to this and it is yet to hit mainstream -- The art itself is stunning but it goes beyond that, the ability to nudge it around and make variations to it is incredible. now add the fact that you can train it with your own content. people are going to go bonkers with this and it's going to open up a lot of debates too.
- egypturnash 4y agoJesus. Well thanks for your contribution to putting the entire creative industry out of work, I guess, little anime girl icon person. Ugh.
- ccity88 4y agoHow very pessimistic. We should never shirk technological progress for fear of upsetting the status quo or established agenda. All of this is only a matter of time away from emerging. Have fun being on the forgotten side of history
- axg11 4y agoA new technology is developed with the potential to make you 100x more efficient at your job. Today, a creative artist can only contribute to a project through a narrow slice. Tomorrow, the same creative artist can single-handedly orchestrate an entire project.
- astrange 4y agoIt's more like there were tasks that were previously so unproductive they couldn't be done at all, and now they're productive enough you might be able to be employed doing them. Automation creates jobs rather than destroying them. What destroys jobs is mainly bad macroeconomic conditions.
- uni_rule 4y agoThis is a force multiplayer. It doesn't take the place of artistic intent, dingus. Besides you can't accomplish much with just "a model". This is an asset generator, hardly a threat to anyone especially when these things will likely need some weight painting to touch up anyway.
- mrtranscendence 4y agoI don’t think the commenter is upset that this particular model will be deployed, putting creative professionals out of work. It’s clearly a janky proof of concept. I think they’re upset about what follow on work could eventually mean.
- Geee 4y agoWould be cool to see it adapted to img2img scenario, using one or more 'seed images'. It would be closer to a standard NERF, but also would be able to imagine novel angles and guidance with prompt.
- etaioinshrdlu 4y agoHow long does it take? 5 hours?
- etaioinshrdlu 4y agoI think it would be interesting to convert to a polygon mesh periodically in-the-loop. It could end up with more precise models.
- xwdv 4y agoFeels like AI generated art is approaching a sort of singularity at this point. Progress getting very exponential.
- antegamisou 4y agoOnly if you have low standards of what constitutes research on artificial intelligence and art perception.
- hwers 4y agoThese things take months and months to train (hardly fast progress). Any new model that’s coming out is generally known in the atmosphere (not unpredictable) and these applications were pretty expected the day stable diffusion came out.
- bpodgursky 4y ago"months and months" At the beginning of this year, most technical people would have told you that graphic design was a decade from being automated, and creative video production more. Now we are at "months and months".
- hwers 4y agoThat this would have come out was totally predictable and expected to most in the know people in the ML world (we basically had proto versions of it summer 2021). Not really the unpredictable trajectory I associate with a singularity
- whazor 4y agoI would love a AI model that does text-to-SVG.
- baxtr 4y agoCan someone explain the significance of this? I am not familiar what DreamFusion is.
- hwers 4y agoOnly downside with this is that each mesh takes like 5 hours to generate (on a v100 too). Obviously it’ll speed up but we’re far from the panacea
- thehappypm 4y agoOmg, imagine how useful this would be for video games or movies. Whipping up an asset in a matter of hours of computer time? Amazing
- verbify 4y agoOr any sort of CAD
- gpm 4y agoMaybe I just don't understand what goes into CAD (probably even), but I don't see it? CAD needs to spit out precise dimensions for things at the end, while this sort of AI is just going to spit out "something more or less like this", it doesn't seem likely to be usable?
- krasin 4y agoA few papers down the road, hopefully. There's no inherent limit for these models to be bad with numbers, as they are now. I do feel that the new-gen mechanical CAD will be based on a dialog between a human and a generational suite. Whether it's based on diffusion or other method - that remains to be seen.
- charlzeee 4y agoI don't think this kind of AI will be that useful for CAD, the time consuming part of CADing is the precise details of things that need to fit together and often iterated on because of various mechanical, electrical and material properties.
- dagw 4y agoA lot of CAD work today is based on parametric design, constraint solvers and letting you CAD software solve various optimisation problems. One of the big problems with this is that the dimensionality and search space involved can be huge. Using AI to narrow the search space that these solvers have to deal with will be a significant and obvious win, and work is already happening in this space. Combine that with an AI that can generate an initial design based on a rough specification, have the user apply some constraints and then iteratively have the AI generate new designs, based on known good designs, that fulfil those constraints could be very powerful.