24 ms·

newb question: Do these models actually learn a 3D representation or do they just learn "something" that is good enough to produce an very convincing impressio

by wiz21c 3y ago

newb question: Do these models actually learn a 3D representation or do they just learn "something" that is good enough to produce an very convincing impression of 3D ? Subquestion: if they don't learn 3D, can we say that models learning a 3D representation first will lead to even better productions ?

astrange 3y ago
> Do these models actually learn a 3D representation or do they just learn "something" that is good enough to produce a very convincing impression of 3D ? The second, but at the limit it's the same thing of course. > Subquestion: if they don't learn 3D, can we say that models learning a 3D representation first will lead to even better productions ? Generally speaking manual feature engineering almost always turns out to be a waste of time if you can just make the model bigger; this is called "the bitter lesson".
l33tman 3y ago
It has been shown that at least still-image generators learn a 3D representation internally and uses it to bootstrap their generation. If you think about it this is the only way they can be so good at shadows and reflections, perspective and lighting etc.
- wiz21c 3y ago
  Could you point at some articles which explores that aspect ? It sounds very interesting. Thanks !
  - l33tman 3y ago
    Please see https://arxiv.org/abs/2306.05720 https://arxiv.org/abs/2306.05720 Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model "... In this work, we investigate a basic interpretability question: does an LDM create and use an internal representation of simple scene geometry? Using linear probes, we find evidence that the internal activations of the LDM encode linear representations of both 3D depth data and a salient-object / background distinction. These representations appear surprisingly early in the denoising process−well before a human can easily make sense of the noisy images. ..."

by wiz21c 3y ago

astrange 3y ago

> Do these models actually learn a 3D representation or do they just learn "something" that is good enough to produce a very convincing impression of 3D ? The second, but at the limit it's the same thing of course. > Subquestion: if they don't learn 3D, can we say that models learning a 3D representation first will lead to even better productions ? Generally speaking manual feature engineering almost always turns out to be a waste of time if you can just make the model bigger; this is called "the bitter lesson".

l33tman 3y ago

It has been shown that at least still-image generators learn a 3D representation internally and uses it to bootstrap their generation. If you think about it this is the only way they can be so good at shadows and reflections, perspective and lighting etc.

wiz21c 3y ago
Could you point at some articles which explores that aspect ? It sounds very interesting. Thanks !
- l33tman 3y ago
  Please see https://arxiv.org/abs/2306.05720 https://arxiv.org/abs/2306.05720 Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model "... In this work, we investigate a basic interpretability question: does an LDM create and use an internal representation of simple scene geometry? Using linear probes, we find evidence that the internal activations of the LDM encode linear representations of both 3D depth data and a salient-object / background distinction. These representations appear surprisingly early in the denoising process−well before a human can easily make sense of the noisy images. ..."