4 ms·

So how do they get the normals, for example? Are they generated by the AI before the image is generated in order to generate the image, and they are just readin

So how do they get the normals, for example? Are they generated by the AI before the image is generated in order to generate the image, and they are just reading them off some internal state?

corysama 3y ago
The training data is just billions of {RGB image, text description} pairs. So, it appears the model figured out how to make the normals as part of the process of making the images. Or, are you asking how the researchers extracted it? > I-LoRA modulates key feature maps to extract intrinsic scene properties such as normals, depth, albedo, and shading, using the models' existing decoders without additional layers, revealing their deep understanding of scene intrinsics.
- auggierose 3y ago
  Yes, that sentence "modulates key feature maps" doesn't tell me anything. What do they mean when saying they extract the normals?
dougmwne 3y ago
Yes, there’s rudimentary evidence that there’s essentially a 3D engine within the model that participates in generating the image. If we could inspect and interpret the whole process it would likely be bizarre and byzantine, like a sort of convergent evolution that independently recreates a tangled spaghetti mess of unreal engine, adobe lightroom, and a physical simulation of a Canon D5.
- qingcharles 3y ago
  Essentially similar perhaps to the 3D engine that a human brain runs that generates a single "3D" image from two 2D cameras (eyes) and fills in missing objects in blind spots, etc.
  - tsimionescu 3y ago
    Note that while having two eyes helps build a more accurate 3D image, people with one eye still see in 3D. Eye movement is at least as important a part of 3D vision as stereoscopy.
    - dougmwne 3y ago
      And apparently 3D renders can be inferred from only 2d images like in these image gen models, so even without video or parallax, brains could probably model the world in 3D.
  - daxfohl 3y ago
    I remember a wittgenstein thing asking to think of a tree or something, and then point to the place in your head where that thought exists. It's kind of like that.

So how do they get the normals, for example? Are they generated by the AI before the image is generated in order to generate the image, and they are just reading them off some internal state?

corysama 3y ago

The training data is just billions of {RGB image, text description} pairs. So, it appears the model figured out how to make the normals as part of the process of making the images. Or, are you asking how the researchers extracted it? > I-LoRA modulates key feature maps to extract intrinsic scene properties such as normals, depth, albedo, and shading, using the models' existing decoders without additional layers, revealing their deep understanding of scene intrinsics.

auggierose 3y ago
Yes, that sentence "modulates key feature maps" doesn't tell me anything. What do they mean when saying they extract the normals?

dougmwne 3y ago

Yes, there’s rudimentary evidence that there’s essentially a 3D engine within the model that participates in generating the image. If we could inspect and interpret the whole process it would likely be bizarre and byzantine, like a sort of convergent evolution that independently recreates a tangled spaghetti mess of unreal engine, adobe lightroom, and a physical simulation of a Canon D5.

qingcharles 3y ago
Essentially similar perhaps to the 3D engine that a human brain runs that generates a single "3D" image from two 2D cameras (eyes) and fills in missing objects in blind spots, etc.
- tsimionescu 3y ago
  Note that while having two eyes helps build a more accurate 3D image, people with one eye still see in 3D. Eye movement is at least as important a part of 3D vision as stereoscopy.
  - dougmwne 3y ago
    And apparently 3D renders can be inferred from only 2d images like in these image gen models, so even without video or parallax, brains could probably model the world in 3D.
- daxfohl 3y ago
  I remember a wittgenstein thing asking to think of a tree or something, and then point to the place in your head where that thought exists. It's kind of like that.