5 ms·

Magic123: One Image to High-Quality 3D Object Generation

220 points by GaggiX 3y ago

ugh123 3y ago
Amazing work! It was be interesting to create a 'spatial viewer' that could navigate around the object, rather than just watching the object rotate in front of you. If theres a background scene, then the object of interest could be extracted and three-dimensionalized, while keeping the background scene flat. Then the viewer could move around the object while maintaining (at least a flat version of) the background.
reidjs 3y ago
Can I use this?
- carbocation 3y ago
  The code is here, so it seems so: https://github.com/guochengqian/Magic123 https://github.com/guochengqian/Magic123 It looks like if you use the full pipeline, it takes 2.5 hours for textual inversion, 40 minutes for coarse estimation, and 20 minutes for fine estimation. For one image, on a 32G V100.
kallistisoft 3y ago
Very impressive work, but this reads more as data compression and retrieval algorithm than a generative model. For all of the examples they're inputting a novel 2D view of a known object and reconstructing the 3D mesh from the 2D/3D training set of the same object. I'd like to see how it performs on a novel object that is similar to one in the training set, but not included in the training data.
- Closi 3y ago
  It sounds to me like the novel view is generated from another algorithm - then their algorithm makes a mesh from these generated novel views to 'improve' the output in a two stage process (coarse -> fine).
q_andrew 3y ago
Certainly a step above what a normal person could do in a 3D modeling program. The interesting thing about this type of stuff is that it would be harder for a professional to use those meshes as a base model; the topology doesn't lend itself to their process. For example -- the banana peels are puffy and two-dimensional. It would have to be completely restructured to be a convincingly peeled banana. So either the generated model has to be a finished product, or they are cost-effectively useless. CAD files are notoriously bad, and those are nice and mathematically easy to break down.
- spookie 3y ago
  Quite true. It's great to have a 3D representation of what you want, however. Gotta say, tools like these or NeRF will only revolutionise 3D modelling if they ever get topology right. That's the hard part.
  - andybak 3y ago
    You're assuming that this is always part of a pipeline that ends with a clean mesh representation that can be rendered using traditional techniques. I think it's equally likely that we'll end up with replacing meshes - or with a hybrid pipeline where non-mesh representations coexist with something else. I am thinking about realtime mainly but I think the same thing might apply to "offline" rendering (does anyone still call it that?)
    - spookie 3y ago
      You might be right. Interesting
- badsectoracula 3y ago
  > So either the generated model has to be a finished product, or they are cost-effectively useless. Not really. Depending on the use case and adequate tools it can be much faster to the alternative of making these manually as meshes and textures. Using the banana as an example, if this can be converted to a volumetric model (voxels) the puffied side can be shaved off using sculpting tools and the model be converted to a mesh much faster than making it from scratch. While the end result wouldn't be good for looking at it up front, it can be perfectly viable for background props in a game, especially something that is viewed from a bird's eye view or a drawn out third person perspective (though even up front it'll look better than what you see in some games[0] - and that is AAA). In fact there have been several games using photogrammetry already to construct 3D models out of taking photos of real places from various angles and converting them to point clouds and then to meshes - which after that they need to be cleaned up by artists. This all takes time, is costly and needs specialized hardware and software and yet developers do it. The linked paper is about a method that significantly lowers those barriers while giving decent results even if they still need to be edited. [0] https://i.imgur.com/0OeNkZu.jpeg https://i.imgur.com/0OeNkZu.jpeg
BudaDude 3y ago
It's pretty cool how quickly we're able to turn a 2D image into a 3D one these days. Although it's not perfect yet, you can see that it doesn't quite capture the depth of the horse and dragon sculpture near the edges. But this technology will definitely improve over time. Now, here's a thought to chew on: once we've mastered 2D to 3D, what's stopping us from exploring 3D to 4D conversion? That would take this to an entirely new level.
- inportb 3y ago
  This project seems to perform some rudimentary 3D to 4D conversion by rotating the object as a function of time. A first step, perhaps, on the path to inferring entire timelines.
Geee 3y ago
Why are researchers focusing on single-image reconstruction? It seems like a party trick which isn't very useful, and pretty much impossible to reconstruct the original object accurately. It would be much more useful if many images from different angles could be used. Somewhat like NERF, but also predict missing views with 2D diffusion. Adding more images would get the model closer to ground truth.
- vlovich123 3y ago
  Because there’s people working on all sorts of different problems and solving a problem in one area can apply better to some problems than others. Not to mention that solution approaches can often cross pollinate. Research is additive not a zero sum thing.
- inconceivable 3y ago
  yeah, it's totally useless.... unless of course, you have only one image of the thing you're trying to model. and getting a second image will take millions of dollars or another trip around the earth from orbit. lmao are you for real?
- zdkl 3y ago
  https://en.wikipedia.org/wiki/Photogrammetry https://en.wikipedia.org/wiki/Photogrammetry https://colmap.github.io/ https://colmap.github.io/ Or, more modern: https://alicevision.org/ https://alicevision.org/
- fxtentacle 3y ago
  They likely focus on the 1 image case because there it's the easiest to show visual progress over the competition. If you have 50+ images, the tech from 10 years ago is already good enough to get Hollywood-quality 3D scans. From what I understand, they use the text keywords detected from the image as guidance and they also apply a loss between the current diffusion state and the source image. In effect, this is stable diffusion for 3D shapes but with clever conditioning. That means this algorithm will also work just fine if you have 2+ input images.
  - 3y ago
lukevp 3y ago
The banana has 2 bananas, and the bird has 2 beaks! Why is that? Does it not know about real life objects and what the image might be representing? Would it be possible to combine this with some other kind of model so it would be able to handle things like that?
- regularfry 3y ago
  https://www.digital-science.com/tldr/article/the-lone-banana-problem-or-the-new-programming-speaking-ai/ https://www.digital-science.com/tldr/article/the-lone-banana...
pdar4123 3y ago
Good lord the horror - devil horse is going to haunt me
- TheCleric 3y ago
  5 legged devil horse.
going_ham 3y ago
NeRFs are cool technology that has its place and use. It is going to help with scene reconstruction and so on. And that is the reason why these CV researchers are flocking into using it. Despite knowing the limitations, they are giving their best to improve this technology. What I believe is NeRF is insufficient to be meaningful on its own. If you look into the architecture from this problem (the paper in the post), it clearly shows that they have some refinement phase going later on. A single RGB 2D to 3D model is such an ill-posed, we have to consider a lot of priors before diving into it. There needs to be more foundational work in this field that can outperform or even improve the NeRF-based techniques. And the current herd mentality of researchers should be changed into exploring the alternatives. There is a reason why expensive automobile companies still rely on the physical modelling of their design. It's hard to simulate the physical conditions only through CAD modelling. Sure NeRFs are cool and they can make impressive results. That doesn't necessarily mean it is the means to an end. Look where rasterization brought us! NeRF is like rasterization. It is going to be used. But highly quality graphics was possible through GI and ray tracing! NeRF needs something equivalent that is physically grounded.
- milkey_mouse 3y ago
  > NeRF needs something equivalent that is physically grounded. Have you seen Plenoxels? https://alexyu.net/plenoxels/ https://alexyu.net/plenoxels/
amelius 3y ago
It would be nice if you could refine the output by adding more images to the input.
robertlagrant 3y ago
Incredible. Although the bear with loops instead of ears is horrifying :) Next step: a few frames of video to produce a fully textured and articulated 3d model?
kristopolous 3y ago
These things are all limited to single objects and can't process say, to be complex, a tea party in a rose garden, right?
blueblisters 3y ago
Interesting work! But given the time and complexity of the generation, I think VSD-loss based implementations have higher fidelity. See https://github.com/thu-ml/prolificdreamer https://github.com/thu-ml/prolificdreamer or https://github.com/threestudio-project/threestudio/tree/main/threestudio https://github.com/threestudio-project/threestudio/tree/main...

220 points by GaggiX 3y ago

Amazing work! It was be interesting to create a 'spatial viewer' that could navigate around the object, rather than just watching the object rotate in front of you. If theres a background scene, then the object of interest could be extracted and three-dimensionalized, while keeping the background scene flat. Then the viewer could move around the object while maintaining (at least a flat version of) the background.

reidjs 3y ago

Can I use this?

carbocation 3y ago
The code is here, so it seems so: https://github.com/guochengqian/Magic123 https://github.com/guochengqian/Magic123 It looks like if you use the full pipeline, it takes 2.5 hours for textual inversion, 40 minutes for coarse estimation, and 20 minutes for fine estimation. For one image, on a 32G V100.

kallistisoft 3y ago

Very impressive work, but this reads more as data compression and retrieval algorithm than a generative model. For all of the examples they're inputting a novel 2D view of a known object and reconstructing the 3D mesh from the 2D/3D training set of the same object. I'd like to see how it performs on a novel object that is similar to one in the training set, but not included in the training data.

Closi 3y ago
It sounds to me like the novel view is generated from another algorithm - then their algorithm makes a mesh from these generated novel views to 'improve' the output in a two stage process (coarse -> fine).

q_andrew 3y ago

Certainly a step above what a normal person could do in a 3D modeling program. The interesting thing about this type of stuff is that it would be harder for a professional to use those meshes as a base model; the topology doesn't lend itself to their process. For example -- the banana peels are puffy and two-dimensional. It would have to be completely restructured to be a convincingly peeled banana. So either the generated model has to be a finished product, or they are cost-effectively useless. CAD files are notoriously bad, and those are nice and mathematically easy to break down.

spookie 3y ago
Quite true. It's great to have a 3D representation of what you want, however. Gotta say, tools like these or NeRF will only revolutionise 3D modelling if they ever get topology right. That's the hard part.
- andybak 3y ago
  You're assuming that this is always part of a pipeline that ends with a clean mesh representation that can be rendered using traditional techniques. I think it's equally likely that we'll end up with replacing meshes - or with a hybrid pipeline where non-mesh representations coexist with something else. I am thinking about realtime mainly but I think the same thing might apply to "offline" rendering (does anyone still call it that?)
  - spookie 3y ago
    You might be right. Interesting
badsectoracula 3y ago
> So either the generated model has to be a finished product, or they are cost-effectively useless. Not really. Depending on the use case and adequate tools it can be much faster to the alternative of making these manually as meshes and textures. Using the banana as an example, if this can be converted to a volumetric model (voxels) the puffied side can be shaved off using sculpting tools and the model be converted to a mesh much faster than making it from scratch. While the end result wouldn't be good for looking at it up front, it can be perfectly viable for background props in a game, especially something that is viewed from a bird's eye view or a drawn out third person perspective (though even up front it'll look better than what you see in some games[0] - and that is AAA). In fact there have been several games using photogrammetry already to construct 3D models out of taking photos of real places from various angles and converting them to point clouds and then to meshes - which after that they need to be cleaned up by artists. This all takes time, is costly and needs specialized hardware and software and yet developers do it. The linked paper is about a method that significantly lowers those barriers while giving decent results even if they still need to be edited. [0] https://i.imgur.com/0OeNkZu.jpeg https://i.imgur.com/0OeNkZu.jpeg

BudaDude 3y ago

It's pretty cool how quickly we're able to turn a 2D image into a 3D one these days. Although it's not perfect yet, you can see that it doesn't quite capture the depth of the horse and dragon sculpture near the edges. But this technology will definitely improve over time. Now, here's a thought to chew on: once we've mastered 2D to 3D, what's stopping us from exploring 3D to 4D conversion? That would take this to an entirely new level.

inportb 3y ago
This project seems to perform some rudimentary 3D to 4D conversion by rotating the object as a function of time. A first step, perhaps, on the path to inferring entire timelines.

Geee 3y ago

Why are researchers focusing on single-image reconstruction? It seems like a party trick which isn't very useful, and pretty much impossible to reconstruct the original object accurately. It would be much more useful if many images from different angles could be used. Somewhat like NERF, but also predict missing views with 2D diffusion. Adding more images would get the model closer to ground truth.

vlovich123 3y ago
Because there’s people working on all sorts of different problems and solving a problem in one area can apply better to some problems than others. Not to mention that solution approaches can often cross pollinate. Research is additive not a zero sum thing.
inconceivable 3y ago
yeah, it's totally useless.... unless of course, you have only one image of the thing you're trying to model. and getting a second image will take millions of dollars or another trip around the earth from orbit. lmao are you for real?
zdkl 3y ago
https://en.wikipedia.org/wiki/Photogrammetry https://en.wikipedia.org/wiki/Photogrammetry https://colmap.github.io/ https://colmap.github.io/ Or, more modern: https://alicevision.org/ https://alicevision.org/
fxtentacle 3y ago
They likely focus on the 1 image case because there it's the easiest to show visual progress over the competition. If you have 50+ images, the tech from 10 years ago is already good enough to get Hollywood-quality 3D scans. From what I understand, they use the text keywords detected from the image as guidance and they also apply a loss between the current diffusion state and the source image. In effect, this is stable diffusion for 3D shapes but with clever conditioning. That means this algorithm will also work just fine if you have 2+ input images.
- 3y ago

lukevp 3y ago

The banana has 2 bananas, and the bird has 2 beaks! Why is that? Does it not know about real life objects and what the image might be representing? Would it be possible to combine this with some other kind of model so it would be able to handle things like that?

regularfry 3y ago
https://www.digital-science.com/tldr/article/the-lone-banana-problem-or-the-new-programming-speaking-ai/ https://www.digital-science.com/tldr/article/the-lone-banana...

pdar4123 3y ago

Good lord the horror - devil horse is going to haunt me

TheCleric 3y ago
5 legged devil horse.

going_ham 3y ago

NeRFs are cool technology that has its place and use. It is going to help with scene reconstruction and so on. And that is the reason why these CV researchers are flocking into using it. Despite knowing the limitations, they are giving their best to improve this technology. What I believe is NeRF is insufficient to be meaningful on its own. If you look into the architecture from this problem (the paper in the post), it clearly shows that they have some refinement phase going later on. A single RGB 2D to 3D model is such an ill-posed, we have to consider a lot of priors before diving into it. There needs to be more foundational work in this field that can outperform or even improve the NeRF-based techniques. And the current herd mentality of researchers should be changed into exploring the alternatives. There is a reason why expensive automobile companies still rely on the physical modelling of their design. It's hard to simulate the physical conditions only through CAD modelling. Sure NeRFs are cool and they can make impressive results. That doesn't necessarily mean it is the means to an end. Look where rasterization brought us! NeRF is like rasterization. It is going to be used. But highly quality graphics was possible through GI and ray tracing! NeRF needs something equivalent that is physically grounded.

milkey_mouse 3y ago
> NeRF needs something equivalent that is physically grounded. Have you seen Plenoxels? https://alexyu.net/plenoxels/ https://alexyu.net/plenoxels/

amelius 3y ago

It would be nice if you could refine the output by adding more images to the input.

robertlagrant 3y ago

Incredible. Although the bear with loops instead of ears is horrifying :) Next step: a few frames of video to produce a fully textured and articulated 3d model?

kristopolous 3y ago

These things are all limited to single objects and can't process say, to be complex, a tea party in a rose garden, right?

blueblisters 3y ago

Interesting work! But given the time and complexity of the generation, I think VSD-loss based implementations have higher fidelity. See https://github.com/thu-ml/prolificdreamer https://github.com/thu-ml/prolificdreamer or https://github.com/threestudio-project/threestudio/tree/main/threestudio https://github.com/threestudio-project/threestudio/tree/main...