5 ms·

The most incredible thing here is that this demonstrates a level of 3D understanding that I didn't believe existed in 2D image models yet. All of the 3D informa

The most incredible thing here is that this demonstrates a level of 3D understanding that I didn't believe existed in 2D image models yet. All of the 3D information in the output was inferred from the training set, which is exclusively uncurated and unsorted 2D still images. No 3D models, no camera parameters, no depth maps. No information about picture content other than a text label (scraped from the web and often incorrect!). From a pile of random undifferentiated images the model has learned the detailed 3D structure and plausible poses and variants of thousands (millions?) of everyday objects. And all we needed to get that 3D information out of the model was the right sampling procedure.

adamredwoods 4y ago
So I wonder if unusual angles that normally do not get photographed will be distorted? For example, underneath a table looking up.
- jacobr1 4y ago
  They reapply noise to the potentially distorted image and then predict the de-noised version like the originally rendered first frame. So the image is at least internally consistent for the frame (to the extend the the system generates consistency whatsoever). The example with a squirrel wearing a hoodie demonstrates an interesting edge case, the "front" of the squirrel (with hoodie over the head) show a normal hooded face as expected, but when you rotate to the "back" you get another face where the hoodie is low over the eyes. Each looks fine in isolation, but in aggregate it seems like we have a two-faced squirrel.
- yarg 4y ago
  It'll be delusions and guesses, rather than distortions. It'll just make up some colours and geometries that don't contradict anything it already knows from the defined perspectives. Or leave it empty.
- poolio 4y ago
  Yes, this is often a problem. We use view-dependent prompts (e.g. "cat wearing sunglasses, back view") but the pretrained 2D model often does not do a good job of interpreting non-canonical views and will put sunglasses on the back of the cats head (as well as the front).
  - salawat 4y ago
    >cat wearing sunglasses, back view") Bad prompt, missing implied antecedent/ambiguous subject... You may want: Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either.
    - thrown_22 4y ago
      >Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either. In short: natural language is not good enough and you need a DSL. If only the last 60 years of language research had warned us of this. Up next: English sentences are ambiguous and need context information to parse correctly. Machine learning community in shambles.
      - salawat 4y ago
        I mean... It kinda did. Sarcasm aside. The trouble is all those darn uninitiated and trying to create a generalized oracle to map their inspecific ramblings to what they mean to free them of having to actually communicate properly... Actually, funnily enough, this has cross section with philosophy in a way most programmers scoff at; but communication is frigging hard, and worse, detecting when someone is trying to get something across, but just needs a nudge in the right direction to be able to find the language to explain it is really damn hard. I run into it every time I get a haircut. I have no idea how to speak their language, so it's always "Uh... A little off the top and rounded at the back, I guess?"
    - 4y ago
- LordDragonfang 4y ago
  Some of the "mesh exports" used as examples on the page actually show this, to some extent. Look specifically at the plush corgi's belly and the weird geometry on the underside of the lemur's book, and to a lesser extent the underside of the bedsheet-ghost.
- anentropic 4y ago
  You can see that in some of these examples e.g. "plush toy of a corgi nurse"
skinner_ 4y ago
Yes, this is amazing. The reason I am a bit less surprised is that I've been playing with textual inversion in the last few days. Just from six photos of me, it can create portraits showing me from any direction. That means it has a very good 3D model of my face, and that information is fully contained in the model plus the CLIP embedding obtained by textual inversion.
poolio 4y ago
Co-author here - we were also surprised :) The breadth of knowledge of the visual world embedded in these 2D models and what they unlock is astounding.
- Vt71fcAqt7 4y ago
  Any word about how Nerf -> marching cubes works? I thought that was still an open problem. Is that another discovery in this research paper?
  - poolio 4y ago
    just seems to work when the 3D model is simple and smooth
- arkis22 4y ago
  the breadth of knowledge of the visual word embedded in these lines and what they unlock is astounding -the point
GistNoesis 4y ago
As far as I understand from a quick read of the paper, the 2D diffusion doesn't have a 3D understanding. It probably have some sort of local neighborhood understanding, aka small geometric transformation of objects map close to each other in the diffusion space (That's why like with latent spaces you can "interpolate" (https://replicate.com/andreasjansson/stable-diffusion-animation https://replicate.com/andreasjansson/stable-diffusion-animat...) in the diffusion space). But that's not really surprising because when you have enough data, even simple clustering methods group objects like faces by the direction they are looking to. With enough views even a simple L2 distance in pixel space allow t-SNE to do that. They are injecting the 3D constraints via the NERF and an optimization process to add the consistency between the frames. It's a deep dream process that optimize by alternating updates for 3D consistency, and updates for text-to-2Dimage correspondence. It's searching for a solution that satisfy these two constraints at the same time. Even though they only need to run a single diffusion step to get the update direction, this optimization process is quite long : 1h30 (but they are not using things like instant Nerf (or even simple voxel grids) ). But this will allow for creation of a dataset of 3D objects with corresponding text, which will then allow to train a diffusion model that will have a 3D understanding and will be able to generate 3D objects directly with a single diffusion process.
- modeless 4y ago
  The model clearly has an understanding of the 3D structure of objects. If it didn't, using it to generate 3D models wouldn't work. The knowledge that the leg bone is connected to the knee bone, etc, isn't coming from NeRF, it's all in the "2D" model. Sure, maybe you could distill that knowledge into a different model architecture that is somehow natively 3D in order to improve the efficiency of sampling. But that's more or less just an optimization. What's interesting is the fact that the knowledge is in there already, learned without 3D input data. Humans don't get natively 3D training data either. We don't have depth sensors or any other natively 3D senses. Our eyes only ever see 2D images and we learn 3D structure from that, so I guess these 2D image models are doing something analagous. Except they don't even have the benefit of stereo images!
  - coolspot 4y ago

adamredwoods 4y ago

So I wonder if unusual angles that normally do not get photographed will be distorted? For example, underneath a table looking up.

jacobr1 4y ago
They reapply noise to the potentially distorted image and then predict the de-noised version like the originally rendered first frame. So the image is at least internally consistent for the frame (to the extend the the system generates consistency whatsoever). The example with a squirrel wearing a hoodie demonstrates an interesting edge case, the "front" of the squirrel (with hoodie over the head) show a normal hooded face as expected, but when you rotate to the "back" you get another face where the hoodie is low over the eyes. Each looks fine in isolation, but in aggregate it seems like we have a two-faced squirrel.
yarg 4y ago
It'll be delusions and guesses, rather than distortions. It'll just make up some colours and geometries that don't contradict anything it already knows from the defined perspectives. Or leave it empty.
poolio 4y ago
Yes, this is often a problem. We use view-dependent prompts (e.g. "cat wearing sunglasses, back view") but the pretrained 2D model often does not do a good job of interpreting non-canonical views and will put sunglasses on the back of the cats head (as well as the front).
- salawat 4y ago
  >cat wearing sunglasses, back view") Bad prompt, missing implied antecedent/ambiguous subject... You may want: Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either.
  - thrown_22 4y ago
    >Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either. In short: natural language is not good enough and you need a DSL. If only the last 60 years of language research had warned us of this. Up next: English sentences are ambiguous and need context information to parse correctly. Machine learning community in shambles.
    - salawat 4y ago
      I mean... It kinda did. Sarcasm aside. The trouble is all those darn uninitiated and trying to create a generalized oracle to map their inspecific ramblings to what they mean to free them of having to actually communicate properly... Actually, funnily enough, this has cross section with philosophy in a way most programmers scoff at; but communication is frigging hard, and worse, detecting when someone is trying to get something across, but just needs a nudge in the right direction to be able to find the language to explain it is really damn hard. I run into it every time I get a haircut. I have no idea how to speak their language, so it's always "Uh... A little off the top and rounded at the back, I guess?"
  - 4y ago
LordDragonfang 4y ago
Some of the "mesh exports" used as examples on the page actually show this, to some extent. Look specifically at the plush corgi's belly and the weird geometry on the underside of the lemur's book, and to a lesser extent the underside of the bedsheet-ghost.
anentropic 4y ago
You can see that in some of these examples e.g. "plush toy of a corgi nurse"

skinner_ 4y ago

Yes, this is amazing. The reason I am a bit less surprised is that I've been playing with textual inversion in the last few days. Just from six photos of me, it can create portraits showing me from any direction. That means it has a very good 3D model of my face, and that information is fully contained in the model plus the CLIP embedding obtained by textual inversion.

poolio 4y ago

Co-author here - we were also surprised :) The breadth of knowledge of the visual world embedded in these 2D models and what they unlock is astounding.

Vt71fcAqt7 4y ago
Any word about how Nerf -> marching cubes works? I thought that was still an open problem. Is that another discovery in this research paper?
- poolio 4y ago
  just seems to work when the 3D model is simple and smooth
arkis22 4y ago
the breadth of knowledge of the visual word embedded in these lines and what they unlock is astounding -the point

GistNoesis 4y ago

As far as I understand from a quick read of the paper, the 2D diffusion doesn't have a 3D understanding. It probably have some sort of local neighborhood understanding, aka small geometric transformation of objects map close to each other in the diffusion space (That's why like with latent spaces you can "interpolate" (https://replicate.com/andreasjansson/stable-diffusion-animation https://replicate.com/andreasjansson/stable-diffusion-animat...) in the diffusion space). But that's not really surprising because when you have enough data, even simple clustering methods group objects like faces by the direction they are looking to. With enough views even a simple L2 distance in pixel space allow t-SNE to do that. They are injecting the 3D constraints via the NERF and an optimization process to add the consistency between the frames. It's a deep dream process that optimize by alternating updates for 3D consistency, and updates for text-to-2Dimage correspondence. It's searching for a solution that satisfy these two constraints at the same time. Even though they only need to run a single diffusion step to get the update direction, this optimization process is quite long : 1h30 (but they are not using things like instant Nerf (or even simple voxel grids) ). But this will allow for creation of a dataset of 3D objects with corresponding text, which will then allow to train a diffusion model that will have a 3D understanding and will be able to generate 3D objects directly with a single diffusion process.

modeless 4y ago
The model clearly has an understanding of the 3D structure of objects. If it didn't, using it to generate 3D models wouldn't work. The knowledge that the leg bone is connected to the knee bone, etc, isn't coming from NeRF, it's all in the "2D" model. Sure, maybe you could distill that knowledge into a different model architecture that is somehow natively 3D in order to improve the efficiency of sampling. But that's more or less just an optimization. What's interesting is the fact that the knowledge is in there already, learned without 3D input data. Humans don't get natively 3D training data either. We don't have depth sensors or any other natively 3D senses. Our eyes only ever see 2D images and we learn 3D structure from that, so I guess these 2D image models are doing something analagous. Except they don't even have the benefit of stereo images!
- coolspot 4y ago