The other day, I was training a de-noising diffusion model on a small enough to over-fit dataset, and watching the diffusion process that generates the image, i

The other day, I was training a de-noising diffusion model on a small enough to over-fit dataset, and watching the diffusion process that generates the image, it really seemed some sense of depth perception has emerged from training. (See the image https://twitter.com/ak92501/status/1475858893561606148 https://twitter.com/ak92501/status/1475858893561606148 for an idea of what a de-noising diffusion is) In essence a de-noising diffusion is reversing the process of adding noise to an image. You learn to tell apart noise from image. And to generate an image you take some random noise and remove a little noise multiple times until you have the image.) Watching the diffusion, the intermediate noisy images it generated seemed to stick out of the frame kinda like how a random-dot auto-stereo-gram would. It is hard to tell for sure because the depth effect could also be coming from the light-effects where bright areas tend to stick out, but the depth it produced made sense and seemed coherent with the scene objects. The thing is, at first order, this effect is not supposed to happen. What the model is learning is trying to remove noise from image, it is not trying to somehow infer some depth about the image, encode it, and passing it to its next iteration so that it can have an easier task. It is not supposed to have some memory like a recurring neural network would. But somehow it seemed to have learned to perceive and encode 3d features and store them inside its weights, so that when it's applied iteratively it converge toward the fixed point that is the real object. That's what I would called emergence as a compression artifact. For this effect to happen the network must have some sort of internal 3d representation of objects, which can happen if it takes less space to compress the object as a 3d object than storing multiple views of the objects. Or maybe I'm just projecting my own sense of perception, and this is just a phantom effect. Maybe consciousness is a phenomenon just like this one.