5 ms·

The model clearly has an understanding of the 3D structure of objects. If it didn't, using it to generate 3D models wouldn't work. The knowledge that the leg bo

The model clearly has an understanding of the 3D structure of objects. If it didn't, using it to generate 3D models wouldn't work. The knowledge that the leg bone is connected to the knee bone, etc, isn't coming from NeRF, it's all in the "2D" model. Sure, maybe you could distill that knowledge into a different model architecture that is somehow natively 3D in order to improve the efficiency of sampling. But that's more or less just an optimization. What's interesting is the fact that the knowledge is in there already, learned without 3D input data. Humans don't get natively 3D training data either. We don't have depth sensors or any other natively 3D senses. Our eyes only ever see 2D images and we learn 3D structure from that, so I guess these 2D image models are doing something analagous. Except they don't even have the benefit of stereo images!

coolspot 4y ago
> Our eyes only ever see 2D images and we learn 3D structure from that I think that would be an oversimplification. We do have some 3D information from focus and eye convergence.
- deltasevennine 4y ago
  its not an over simplification. The extra information from convergence is negligible... our eyes derive virtually identical information when looking at flat 2D pictures of 3D scenes. Evidence of this is everywhere in pictures.
  - F2hP18Foam 4y ago
    I'm sure people born with one eye are perfectly able to understand the concept of 3D
    - deltasevennine 4y ago
      Dunno if you fully read my comment. But we're in agreement.
      - F2hP18Foam 4y ago
        You're right, we are in agreement.
ShamelessC 4y ago
The model is trained on discretized photons (pixels). Doing so may have encouraged the model to learn features related to depth.
poolio 4y ago
I think you're both right! It is incredible that the 2D model knows enough about the visual world to produce many objects from all angles, but the 3D model is essential for gluing these views together, and in some ways can fill in the gaps the 2D model doesn't know about. Imagine just taking a huge collection of photographs of an object. While there is enough information in those photos to reconstruct the 3D object, I wouldn't personally call that collection of images "an understanding of 3D." In our case, the diffusion model is the collection of photos and the NeRF model + optimization procedure is what figures out how all those photos can be related to a shared underlying 3D representation. - ben p (author)
sandgiant 4y ago
I think this is a misunderstanding of how these models work. The model does not understand anything at all. It's computing correlations. I could spend all day computing correlation without ever understanding what the correlations correspond to in the physical world. The correlations could still amount to a useful description of some physical phenomenon, but their interpretation requires much more than just the ability to compute correlations in some limited space. If we elevate "correlations" to mean "understanding", we're quickly going to run out of words.
- cornel_io 4y ago
  Your brain merely computes correlations. Does that make it any less intelligent? At some point we have to accept that when you layer simple ops on top of simple ops enough times you get complex behavior.
psychoslave 4y ago
>The model clearly has an understanding of the 3D structure of objects. And the submarine clearly can swim! :D The model encompasses 3D information, but a model is not a thinking entity able to understand anything. Well, it might be argued that saying that the "model understands" is a metaphor, and used as such OK, fine. But that’s it. Analogies rarely scale well. As for humans, have you ever heard of proprioception? If that is not "native 3D sensors", I have no idea of what a 3D sense might be.
- pvillano 4y ago
  https://www.reddit.com/r/Damnthatsinteresting/comments/xrxezn/machine_3d_sculpture/ https://www.reddit.com/r/Damnthatsinteresting/comments/xrxez... A machine with 3d understanding from 2d samples predates computers

coolspot 4y ago

> Our eyes only ever see 2D images and we learn 3D structure from that I think that would be an oversimplification. We do have some 3D information from focus and eye convergence.

deltasevennine 4y ago
its not an over simplification. The extra information from convergence is negligible... our eyes derive virtually identical information when looking at flat 2D pictures of 3D scenes. Evidence of this is everywhere in pictures.
- F2hP18Foam 4y ago
  I'm sure people born with one eye are perfectly able to understand the concept of 3D
  - deltasevennine 4y ago
    Dunno if you fully read my comment. But we're in agreement.
    - F2hP18Foam 4y ago
      You're right, we are in agreement.

ShamelessC 4y ago

The model is trained on discretized photons (pixels). Doing so may have encouraged the model to learn features related to depth.

poolio 4y ago

I think you're both right! It is incredible that the 2D model knows enough about the visual world to produce many objects from all angles, but the 3D model is essential for gluing these views together, and in some ways can fill in the gaps the 2D model doesn't know about. Imagine just taking a huge collection of photographs of an object. While there is enough information in those photos to reconstruct the 3D object, I wouldn't personally call that collection of images "an understanding of 3D." In our case, the diffusion model is the collection of photos and the NeRF model + optimization procedure is what figures out how all those photos can be related to a shared underlying 3D representation. - ben p (author)

sandgiant 4y ago

I think this is a misunderstanding of how these models work. The model does not understand anything at all. It's computing correlations. I could spend all day computing correlation without ever understanding what the correlations correspond to in the physical world. The correlations could still amount to a useful description of some physical phenomenon, but their interpretation requires much more than just the ability to compute correlations in some limited space. If we elevate "correlations" to mean "understanding", we're quickly going to run out of words.

cornel_io 4y ago
Your brain merely computes correlations. Does that make it any less intelligent? At some point we have to accept that when you layer simple ops on top of simple ops enough times you get complex behavior.

psychoslave 4y ago

>The model clearly has an understanding of the 3D structure of objects. And the submarine clearly can swim! :D The model encompasses 3D information, but a model is not a thinking entity able to understand anything. Well, it might be argued that saying that the "model understands" is a metaphor, and used as such OK, fine. But that’s it. Analogies rarely scale well. As for humans, have you ever heard of proprioception? If that is not "native 3D sensors", I have no idea of what a 3D sense might be.

pvillano 4y ago
https://www.reddit.com/r/Damnthatsinteresting/comments/xrxezn/machine_3d_sculpture/ https://www.reddit.com/r/Damnthatsinteresting/comments/xrxez... A machine with 3d understanding from 2d samples predates computers