4 ms·

The current image generators certainly know about 3D internally. It's not put there explicitly but the wealth of knowledge they possess and the perfect lighting

The current image generators certainly know about 3D internally. It's not put there explicitly but the wealth of knowledge they possess and the perfect lighting and shadowing and material reflections compared to the tiny amount of weights strongly suggest they have proper representations (just like humans by the way). Even a small model in early training learns to do proper shadowing and lighting and mostly makes mistakes in the 3D high-level object space, like wrong number of wheels or legs or fingers and stuff like that.

teddyh 4y ago
> It's not put there explicitly but the wealth of knowledge they possess and the perfect lighting and shadowing and material reflections compared to the tiny amount of weights strongly suggest they have proper representations I don’t think they do, but I can’t prove it, either way. > (just like humans by the way). Oh, I don’t think humans have it either (mostly), since humans don’t have 3D vision; humans have the equivalent of wiggle stereoscopy¹, which gives some hints of depth, and humans have the additional advantage of the time dimension. People do have some intellectual capacity to reason about 3-dimensional shapes, and some people can even rotate things in their heads with ease. Blind people might have it too, since their concept of the world was not created by this pseudo-3D visual input. But mostly, people don’t think in 3D. I think we can see this by looking at drawings by children. Child drawings are dominated by the concepts important to children: Faces, hands, etc. Concepts, not actual images. And it’s all in 2D, as was the majority of art for much of history. 1. https://en.wikipedia.org/wiki/Wiggle_stereoscopy https://en.wikipedia.org/wiki/Wiggle_stereoscopy
- l33tman 4y ago
  I have to strongly disagree with you on this, I can walk around in my (small) city in my mind in 3D without any problems. Anybody who's ever had a dream would concur as well. It's absolutely not true that humans can't think in 3D. The sensory input modalities and specifics do not control or limit the internal representations; a sufficiently capable neural network will extract the most efficient rep to predict the input data, which is moving 3D objects. That it's difficult to "render" this to 2D by painting is not very surprising.
  - teddyh 4y ago
    > I have to strongly disagree with you on this, I can walk around in my (small) city in my mind in 3D without any problems. Anybody who's ever had a dream would concur as well. It's absolutely not true that humans can't think in 3D. Well, some people reportedly can, but it’s rare. You say you can imagine walking around your small city, and infer that you have the city conceptualized as a 3D object, and can rotate it at will, in any direction. But how do we know that you simply have not memorized your admittedly small city? Can you imagine what going around the city upside down (i.e. like walking on your hands) would look like, with ease? It should be just a simple rotation, right?
npteljes 4y ago
I don't think that they consider 3d too much. Consider the "Girl with a Pearl Earring" image, that's meant to represent something that's state of the art: https://openai.com/blog/dall-e-introducing-outpainting/ https://openai.com/blog/dall-e-introducing-outpainting/ The image might look consistent at first sight, but if you look closer, the dimensions are all over the place. They reflect the method by which they were created: reproducing observed patterns without deeper understanding.
- l33tman 4y ago
  You can find lots of buggy images with bugs in all possible ways. But the vast majority of the images have consistent shading, perspective, and even global illumination. Some of it can be done in "screen space" (as in some standard game 2D post-processing filters) but lots of them couldn't work this way. At some low level, pixel patterns are rendered and I guess you could say that this is "reproducing observed patterns". Would you say that about a 3D game engine as well that does the same when it textures local regions of pixels? A network is layered for this particular reason. The lowest layer will have less "understanding" than the higher layers.
- astrange 4y ago
  If you ask for something "reflective" or "in a hall of mirrors" it tends to work, though it often seems like a screen-space effect as it tends to draw the front of objects rather than their back in the reflection. https://labs.openai.com/s/OJEkBZSwagsCakHiE9QNdV92 https://labs.openai.com/s/OJEkBZSwagsCakHiE9QNdV92
  - npteljes 4y ago
    The perspective barely changes in the sphere. I'm not convinced that the AI implied 3d, lighting/shadows, and reflections from the dataset which is supposed to be, at most, just 2d projections of 3d environments. Maybe in the future when we feed it 3d (meta) information also, then we can expect some magic, but until then, I don't think so.

teddyh 4y ago

> It's not put there explicitly but the wealth of knowledge they possess and the perfect lighting and shadowing and material reflections compared to the tiny amount of weights strongly suggest they have proper representations I don’t think they do, but I can’t prove it, either way. > (just like humans by the way). Oh, I don’t think humans have it either (mostly), since humans don’t have 3D vision; humans have the equivalent of wiggle stereoscopy¹, which gives some hints of depth, and humans have the additional advantage of the time dimension. People do have some intellectual capacity to reason about 3-dimensional shapes, and some people can even rotate things in their heads with ease. Blind people might have it too, since their concept of the world was not created by this pseudo-3D visual input. But mostly, people don’t think in 3D. I think we can see this by looking at drawings by children. Child drawings are dominated by the concepts important to children: Faces, hands, etc. Concepts, not actual images. And it’s all in 2D, as was the majority of art for much of history. 1. https://en.wikipedia.org/wiki/Wiggle_stereoscopy https://en.wikipedia.org/wiki/Wiggle_stereoscopy

l33tman 4y ago
I have to strongly disagree with you on this, I can walk around in my (small) city in my mind in 3D without any problems. Anybody who's ever had a dream would concur as well. It's absolutely not true that humans can't think in 3D. The sensory input modalities and specifics do not control or limit the internal representations; a sufficiently capable neural network will extract the most efficient rep to predict the input data, which is moving 3D objects. That it's difficult to "render" this to 2D by painting is not very surprising.
- teddyh 4y ago
  > I have to strongly disagree with you on this, I can walk around in my (small) city in my mind in 3D without any problems. Anybody who's ever had a dream would concur as well. It's absolutely not true that humans can't think in 3D. Well, some people reportedly can, but it’s rare. You say you can imagine walking around your small city, and infer that you have the city conceptualized as a 3D object, and can rotate it at will, in any direction. But how do we know that you simply have not memorized your admittedly small city? Can you imagine what going around the city upside down (i.e. like walking on your hands) would look like, with ease? It should be just a simple rotation, right?

npteljes 4y ago

I don't think that they consider 3d too much. Consider the "Girl with a Pearl Earring" image, that's meant to represent something that's state of the art: https://openai.com/blog/dall-e-introducing-outpainting/ https://openai.com/blog/dall-e-introducing-outpainting/ The image might look consistent at first sight, but if you look closer, the dimensions are all over the place. They reflect the method by which they were created: reproducing observed patterns without deeper understanding.

l33tman 4y ago
You can find lots of buggy images with bugs in all possible ways. But the vast majority of the images have consistent shading, perspective, and even global illumination. Some of it can be done in "screen space" (as in some standard game 2D post-processing filters) but lots of them couldn't work this way. At some low level, pixel patterns are rendered and I guess you could say that this is "reproducing observed patterns". Would you say that about a 3D game engine as well that does the same when it textures local regions of pixels? A network is layered for this particular reason. The lowest layer will have less "understanding" than the higher layers.
astrange 4y ago
If you ask for something "reflective" or "in a hall of mirrors" it tends to work, though it often seems like a screen-space effect as it tends to draw the front of objects rather than their back in the reflection. https://labs.openai.com/s/OJEkBZSwagsCakHiE9QNdV92 https://labs.openai.com/s/OJEkBZSwagsCakHiE9QNdV92
- npteljes 4y ago
  The perspective barely changes in the sphere. I'm not convinced that the AI implied 3d, lighting/shadows, and reflections from the dataset which is supposed to be, at most, just 2d projections of 3d environments. Maybe in the future when we feed it 3d (meta) information also, then we can expect some magic, but until then, I don't think so.