5 ms·

An algorithm that recreates 3D objects from tiny 2D images

161 points by janober 9y ago

Stanleyc23 9y ago
This paper from a while back acomplishes this too. https://github.com/chrischoy/3D-R2N2 https://github.com/chrischoy/3D-R2N2
- runesoerensen 9y ago
  Thought of that paper too but didn't remember the name, thanks for sharing! The new paper also reference 3D-R2N2 several times and seems heavily inspired/adapted from it.
EamonnMR 9y ago
Could be interesting to see it run on old game sprites. The Starcraft Remastered team could have used this, for example.
- figgis 9y ago
  How would copyright and licenses apply to something like this?
  - jackhack 9y ago
    presumably covered under the common "derivative works" clause
AndrewKemendo 9y ago
I'm very excited about the potential for extracting voxel specific or procedural generated 3D objects from a low number of 2D images - primarily with a combination of Semantic Segmentation and some form of MVS. The results of papers like this and other similar ones are great progress.
- Aron 9y ago
  Seems like Google Earth is doing this yeah?
  - dharma1 9y ago
    for the 3D Google Earth views? It's photogrammetry from a 5 camera rig on a low flying plane https://youtu.be/suo_aUTUpps?t=2m53s https://youtu.be/suo_aUTUpps?t=2m53s
    - Aron 9y ago
      Alright pretty old school. As an aside, one of the first things I did with my Oculus was pull up Google Earth and lay down in Yosemite valley which is feature around 5 minutes in this vid. Cheers.
- deleted 9y ago
  [deleted]
ricardobeat 9y ago
Very interesting tech, though the article makes a piss poor job of explaining anything about how it works. Rendering surfaces is how literally 99.9% of 3D graphics work, not a "breakthrough" of any sorts.
- yorwba 9y ago
  The abstract is much better. At least they linked to it in the article. https://arxiv.org/abs/1704.00710 https://arxiv.org/abs/1704.00710
- averagewall 9y ago
  I think it describes it quite well as long as you assume the problem of making a 3D model from a picture is already a solved but computationally expensive problem. That's the part it doesn't go into but that's not what this research is about. Though I think most readers will still be interested in that - I didn't know it had been solved already. It's not making a surface mesh, it's putting voxels on the surface. Again, it doesn't make that distinction very clear up front so I guess it's not targeted at people familiar with traditional 3D surface meshing.
- jordancampbell 9y ago
  It's inferring the depth of all the points on the surface, including those it can't see in the image. This isn't surface rendering, it's depth estimation.
- khedoros1 9y ago
  99.9% of 3D graphics: "Computer, here's a definition of some 3D surfaces and their properties. Produce a 2D projection." This: "Computer, here's a 2D projection. Infer the likely 3D data that would produce the projection."
  - ricardobeat 9y ago
    The comparison the article makes is against rendering a full voxel volume, as if occlusion was a great insight.
13of40 9y ago
I kind of got lost on the first sentence: "With a lifetime of observing the world informing our perceptions, we’re all pretty good at inferring the overall shape of something we only see from the side, or for a brief moment. Computers, however, are just plain bad at it." I'm good at inferring the shape of something from a side view well enough that I can use it in some kludgy mental models, but down the page they're essentially asking the computer to look at something from the side and render a precise(ish) 3-D mechanical model of the object from memory. For example, I have what I think is a pretty good idea of what a Boeing 737 looks like, but if you asked me to draw it on a piece of paper it would look like a kinder gardener did it. What I'm good at is boiling down features and distinguishing a 737 from an Oscar Meyer wiener truck. Drawing a scale-accurate picture of it is a job for artists and savants.
- khedoros1 9y ago
  The computer's a savant at precise drawing, but the visualization isn't the interesting part. The interesting part is having the computer look at a 2D picture and come up with a "kludgy mental model" of the 3D shape represented by the picture. From the example images, it infers information more accurately than I would've expected.
  - jackhack 9y ago
    I can't help but wonder what would result from an Escher drawing.
AliAdams 9y ago
I'm looking at the second example from the image in the article - the blue plane - and can't work out how the algorithm could possibly infer a second wing from that picture.
- pasta 9y ago
  It looks like the algorithm expects symmetry.
- unkown-unknowns 9y ago
  Maybe it involves machine learning also and it is able to tell that it's looking at an aeroplane?
- johndough 9y ago
  Presumably the neural network was trained on a dataset where all planes had two wings, so it will predict planes with two wings.
  - TuringTest 9y ago
    Yeah, there's definitely some previous knowledge of the kinds of objects it's inferring, which is used to deduce the parts. Look at the chairs' legs, where it's transforming the flat base of the rotating chair into something that looks as wheels, and where it's completely missing the bars connecting the legs in the second tall chair.
pasta 9y ago
Well I'm sorry but I think more is going on than the article describes. For example the red pickup truck. There is no way the algorithm could create that model from that image only (depth of the trunk). So my guess is that they use the tiny picture to search a database for similar pictures and then create a model with all that data.
- middleman90 9y ago
  I agree. I think that the AI has knowledge of those objects. The fact that all the objects are symmetric (at least in the examples) helps a lot.
- blauditore 9y ago
  The linked paper talks about CNNs in its abstract, so those were probably trained with such samples and thus have that knowledge baked in.
- 31reasons 9y ago
  If you see a photo of an object that you have never seen in your life and have no analogy to draw from, it will indeed be very hard for you to accurately guess its volume. However as humans we come across millions of objects to learn from. Same way you can train a neural network to learn from millions of objects and that would enable them to make similar guesses. If this algorithm manages to be as good as humans in estimating volumes of everyday objects, I can see it being helpful in Self Driving Cars.
vbuwivbiu 9y ago
want to see results from Necker cube
edanm 9y ago
This article is confusingly written. Its explanations sound silly to anyone even a little familiar with 3d, since it spends the first half of the article explaining that this "breakthrough" is "computationally clever and forehead-slappingly simple", the breakthrough being that you can represent things as surface-models instead of voxels. Well, surface rendering is how almost all 3d work is done already, that's definitely not a breakthrough. You can probably spend an entire career never dealing with voxels. What this article never mentions (surprisingly) is that this paper is about neural networks. I'm not an expert, but as I understand the article, voxel representations have been the standard specifically when building neural networks to turn 2d images into 3d. The main idea is that you can build a network that only renders high-resolution voxels when close to the surface of the model, and renders very low-resolution voxels everywhere else (say, on the inside of the model). This means you can both represent much larger models memory-wise, but also that you're not having to run the NN computations on voxels that are not likely to change, since everything "inside" of the model you probably guessed correctly pretty quickly. Here's the paper's abstract, much better at explaining itself than the article: "Recently, Convolutional Neural Networks have shown promising results for 3D geometry prediction. They can make predictions from very little input data such as for example a single color image, depth map or a partial 3D volume. A major limitation of such approaches is that they only predict a coarse resolution voxel grid, which does not capture the surface of the objects well. We propose a general framework, called hierarchical surface prediction (HSP), which facilitates prediction of high resolution voxel grids. The main insight is that it is sufficient to predict high resolution voxels around the predicted surfaces. The exterior and interior of the objects can be represented with coarse resolution voxels. This allows us to predict significantly higher resolution voxel grids around the surface, from which triangle meshes can be extracted. Our approach is general and not dependent on a specific input type. In our experiments we show results for geometry prediction from color images, depth images and shape completion from partial voxel grids. Our analysis shows that the network is able to predict the surface more accurately than a low resolution prediction."
- xg15 9y ago
  I was stumped by that as well. It spends half the text explaining the well-known parts (surface models) and then mentions the actual contribution completely in passing: > So first his system renders a 3D reconstruction of the 2D image in very low resolution [...] Next, do a higher-resolution render of the area you kept. Where apparently the author seems to think producing a 3D reconstruction of a 2D object is trivial even though that's what the paper is about.
fischerq 9y ago
As lots of people already pointed out: there is more going on than just a smart upsampling technique. See section 4 of the paper - they train on a set of 3d models from some 'ShapeNetCore' dataset, from which they generate sample inputs (renderings of the model with randomized viewpoint and lighting) and corresponding target outputs (voxelized model). They train specialized networks for different classes of objects 'aeroplanes, chairs and car', so reconstructing on all classes at the same time probably still has some issues. An interesting point about their use of this coarse-to-fine progression that the article omits: they do the same trick for training their net - first train to predict the coarse voxels, and when those work start predicting the next level.
ealloc 9y ago
"Single Particle Reconstruction" for Cryo-EM reconstruction algorithm, anyone? SPR is an algorithm already used to reconstruct 3d objects from a set of 2d images produced by an electron microscope, typically of a protein on a flat surface. I haven't read the paper here, but I suspect a key part of the algorithm is that it can only reconstruct objects with symmetry planes: The airplane, chair, car are all symmetric across an axis. This greatly constrains the possibilities the algorithm has to search through. In em-reconstruction the user often specifies what they think the symmetries are beforehand.

161 points by janober 9y ago

This paper from a while back acomplishes this too. https://github.com/chrischoy/3D-R2N2 https://github.com/chrischoy/3D-R2N2

runesoerensen 9y ago
Thought of that paper too but didn't remember the name, thanks for sharing! The new paper also reference 3D-R2N2 several times and seems heavily inspired/adapted from it.

EamonnMR 9y ago

Could be interesting to see it run on old game sprites. The Starcraft Remastered team could have used this, for example.

figgis 9y ago
How would copyright and licenses apply to something like this?
- jackhack 9y ago
  presumably covered under the common "derivative works" clause

AndrewKemendo 9y ago

I'm very excited about the potential for extracting voxel specific or procedural generated 3D objects from a low number of 2D images - primarily with a combination of Semantic Segmentation and some form of MVS. The results of papers like this and other similar ones are great progress.

Aron 9y ago
Seems like Google Earth is doing this yeah?
- dharma1 9y ago
  for the 3D Google Earth views? It's photogrammetry from a 5 camera rig on a low flying plane https://youtu.be/suo_aUTUpps?t=2m53s https://youtu.be/suo_aUTUpps?t=2m53s
  - Aron 9y ago
    Alright pretty old school. As an aside, one of the first things I did with my Oculus was pull up Google Earth and lay down in Yosemite valley which is feature around 5 minutes in this vid. Cheers.
deleted 9y ago
[deleted]

ricardobeat 9y ago

Very interesting tech, though the article makes a piss poor job of explaining anything about how it works. Rendering surfaces is how literally 99.9% of 3D graphics work, not a "breakthrough" of any sorts.

yorwba 9y ago
The abstract is much better. At least they linked to it in the article. https://arxiv.org/abs/1704.00710 https://arxiv.org/abs/1704.00710
averagewall 9y ago
I think it describes it quite well as long as you assume the problem of making a 3D model from a picture is already a solved but computationally expensive problem. That's the part it doesn't go into but that's not what this research is about. Though I think most readers will still be interested in that - I didn't know it had been solved already. It's not making a surface mesh, it's putting voxels on the surface. Again, it doesn't make that distinction very clear up front so I guess it's not targeted at people familiar with traditional 3D surface meshing.
jordancampbell 9y ago
It's inferring the depth of all the points on the surface, including those it can't see in the image. This isn't surface rendering, it's depth estimation.
khedoros1 9y ago
99.9% of 3D graphics: "Computer, here's a definition of some 3D surfaces and their properties. Produce a 2D projection." This: "Computer, here's a 2D projection. Infer the likely 3D data that would produce the projection."
- ricardobeat 9y ago
  The comparison the article makes is against rendering a full voxel volume, as if occlusion was a great insight.

13of40 9y ago

I kind of got lost on the first sentence: "With a lifetime of observing the world informing our perceptions, we’re all pretty good at inferring the overall shape of something we only see from the side, or for a brief moment. Computers, however, are just plain bad at it." I'm good at inferring the shape of something from a side view well enough that I can use it in some kludgy mental models, but down the page they're essentially asking the computer to look at something from the side and render a precise(ish) 3-D mechanical model of the object from memory. For example, I have what I think is a pretty good idea of what a Boeing 737 looks like, but if you asked me to draw it on a piece of paper it would look like a kinder gardener did it. What I'm good at is boiling down features and distinguishing a 737 from an Oscar Meyer wiener truck. Drawing a scale-accurate picture of it is a job for artists and savants.

khedoros1 9y ago
The computer's a savant at precise drawing, but the visualization isn't the interesting part. The interesting part is having the computer look at a 2D picture and come up with a "kludgy mental model" of the 3D shape represented by the picture. From the example images, it infers information more accurately than I would've expected.
- jackhack 9y ago
  I can't help but wonder what would result from an Escher drawing.

AliAdams 9y ago

I'm looking at the second example from the image in the article - the blue plane - and can't work out how the algorithm could possibly infer a second wing from that picture.

pasta 9y ago
It looks like the algorithm expects symmetry.
unkown-unknowns 9y ago
Maybe it involves machine learning also and it is able to tell that it's looking at an aeroplane?
johndough 9y ago
Presumably the neural network was trained on a dataset where all planes had two wings, so it will predict planes with two wings.
- TuringTest 9y ago
  Yeah, there's definitely some previous knowledge of the kinds of objects it's inferring, which is used to deduce the parts. Look at the chairs' legs, where it's transforming the flat base of the rotating chair into something that looks as wheels, and where it's completely missing the bars connecting the legs in the second tall chair.

pasta 9y ago

Well I'm sorry but I think more is going on than the article describes. For example the red pickup truck. There is no way the algorithm could create that model from that image only (depth of the trunk). So my guess is that they use the tiny picture to search a database for similar pictures and then create a model with all that data.

middleman90 9y ago
I agree. I think that the AI has knowledge of those objects. The fact that all the objects are symmetric (at least in the examples) helps a lot.
blauditore 9y ago
The linked paper talks about CNNs in its abstract, so those were probably trained with such samples and thus have that knowledge baked in.
31reasons 9y ago
If you see a photo of an object that you have never seen in your life and have no analogy to draw from, it will indeed be very hard for you to accurately guess its volume. However as humans we come across millions of objects to learn from. Same way you can train a neural network to learn from millions of objects and that would enable them to make similar guesses. If this algorithm manages to be as good as humans in estimating volumes of everyday objects, I can see it being helpful in Self Driving Cars.

vbuwivbiu 9y ago

want to see results from Necker cube

edanm 9y ago

This article is confusingly written. Its explanations sound silly to anyone even a little familiar with 3d, since it spends the first half of the article explaining that this "breakthrough" is "computationally clever and forehead-slappingly simple", the breakthrough being that you can represent things as surface-models instead of voxels. Well, surface rendering is how almost all 3d work is done already, that's definitely not a breakthrough. You can probably spend an entire career never dealing with voxels. What this article never mentions (surprisingly) is that this paper is about neural networks. I'm not an expert, but as I understand the article, voxel representations have been the standard specifically when building neural networks to turn 2d images into 3d. The main idea is that you can build a network that only renders high-resolution voxels when close to the surface of the model, and renders very low-resolution voxels everywhere else (say, on the inside of the model). This means you can both represent much larger models memory-wise, but also that you're not having to run the NN computations on voxels that are not likely to change, since everything "inside" of the model you probably guessed correctly pretty quickly. Here's the paper's abstract, much better at explaining itself than the article: "Recently, Convolutional Neural Networks have shown promising results for 3D geometry prediction. They can make predictions from very little input data such as for example a single color image, depth map or a partial 3D volume. A major limitation of such approaches is that they only predict a coarse resolution voxel grid, which does not capture the surface of the objects well. We propose a general framework, called hierarchical surface prediction (HSP), which facilitates prediction of high resolution voxel grids. The main insight is that it is sufficient to predict high resolution voxels around the predicted surfaces. The exterior and interior of the objects can be represented with coarse resolution voxels. This allows us to predict significantly higher resolution voxel grids around the surface, from which triangle meshes can be extracted. Our approach is general and not dependent on a specific input type. In our experiments we show results for geometry prediction from color images, depth images and shape completion from partial voxel grids. Our analysis shows that the network is able to predict the surface more accurately than a low resolution prediction."

xg15 9y ago
I was stumped by that as well. It spends half the text explaining the well-known parts (surface models) and then mentions the actual contribution completely in passing: > So first his system renders a 3D reconstruction of the 2D image in very low resolution [...] Next, do a higher-resolution render of the area you kept. Where apparently the author seems to think producing a 3D reconstruction of a 2D object is trivial even though that's what the paper is about.

fischerq 9y ago

As lots of people already pointed out: there is more going on than just a smart upsampling technique. See section 4 of the paper - they train on a set of 3d models from some 'ShapeNetCore' dataset, from which they generate sample inputs (renderings of the model with randomized viewpoint and lighting) and corresponding target outputs (voxelized model). They train specialized networks for different classes of objects 'aeroplanes, chairs and car', so reconstructing on all classes at the same time probably still has some issues. An interesting point about their use of this coarse-to-fine progression that the article omits: they do the same trick for training their net - first train to predict the coarse voxels, and when those work start predicting the next level.

ealloc 9y ago

"Single Particle Reconstruction" for Cryo-EM reconstruction algorithm, anyone? SPR is an algorithm already used to reconstruct 3d objects from a set of 2d images produced by an electron microscope, typically of a protein on a flat surface. I haven't read the paper here, but I suspect a key part of the algorithm is that it can only reconstruct objects with symmetry planes: The airplane, chair, car are all symmetric across an axis. This greatly constrains the possibilities the algorithm has to search through. In em-reconstruction the user often specifies what they think the symmetries are beforehand.