4 ms·

It's interesting how completely this fails at reflections. It keeps forgetting that the reflection of a mountain range can't just be a different mountain range

It's interesting how completely this fails at reflections. It keeps forgetting that the reflection of a mountain range can't just be a different mountain range with a somewhat similar shape.

amelius 4y ago
How good is DALL-E at completing a picture of a face when (say) the left half of it has been masked out? Is the resulting face symmetrical? If so, why doesn't the same approach work with reflections?
- adgjlsfhk1 4y ago
  I think faces are much easier than reflections for a few reasons. 1. faces aren't perfectly symmetric. There is a bunch of leeway on the small scale 2. faces can be oriented to make lack of symmetry harder to see (most photos aren't people perfectly facing the camera) 3. faces can be parametrized with relatively few parameters. You need correct position and sizing for eyes, nose, mouth, eyebrows and ears. If you get those right, everything else is doable based only on local information. For a mountain range reflection, you need to transfer a ton of exact information across the image (and that information can't easily be embedded in 2d. to know the reflection, you have to have a 3d map).
  - notahacker 4y ago
    Eyeballs appearing to point in different directions and really asymmetric ears are among the common 'tells' of AI generation in a lot of models too. I think the biggest challenge is that unlike facial symmetry, reflected mountains in training sets are often significantly perturbed by ripples and objects in the water. So tones in the water broadly matching the tones of the mountains above but with different texture is "good enough" for the model, and it doesn't catch finer detail mismatches in shapes of edges in the same way it catches finer detail mismatches in the shapes of eyes, which almost invariably have the same texture and similar lighting to their compatriot in images.
- dahart 4y ago
  Faces aren’t perfectly symmetrical though, while mirror reflections are physics. The degree to which the landscape reflection symmetries are off in the article might be acceptable when generating faces. One possible reason the same approach might not work in both cases could be that DALLE’s training set has more faces than landscapes with still water lakes, it’d have to learn these two symmetry constraints through enough clear examples in both cases, and no counter-examples. I can imagine counter-examples for both faces and landscapes that could potentially inhibit learning symmetry, like faces from an angle or side view, or landscapes with an occluder, but I have no idea how often these occur in DALLE’s training set. Getting this right means that DALLE somehow needs to learn the concept and the symmetry constraint from all the imagery, it has to understand the world a little, not just have pixels to borrow. It’s a hard problem, and at some level surprising how well it’s doing with the landscapes.
deleted 4y ago
[deleted]
coding123 4y ago
It's actually really interesting that the mirror image is that close though. The AI could have seen the water and just saw that there is mountainous shapes, but the AI at least noticed that it's similar to the above one - which tells you how deep these nets go. Perhaps for it to know that it's a nearly perfect mirror image it will need more depth.
tauntz 4y ago
I had to try quite hard to spot issues with the reflections after reading the comments here. Maybe I'm wired differently

It's interesting how completely this fails at reflections. It keeps forgetting that the reflection of a mountain range can't just be a different mountain range with a somewhat similar shape.

amelius 4y ago

How good is DALL-E at completing a picture of a face when (say) the left half of it has been masked out? Is the resulting face symmetrical? If so, why doesn't the same approach work with reflections?

adgjlsfhk1 4y ago
I think faces are much easier than reflections for a few reasons. 1. faces aren't perfectly symmetric. There is a bunch of leeway on the small scale 2. faces can be oriented to make lack of symmetry harder to see (most photos aren't people perfectly facing the camera) 3. faces can be parametrized with relatively few parameters. You need correct position and sizing for eyes, nose, mouth, eyebrows and ears. If you get those right, everything else is doable based only on local information. For a mountain range reflection, you need to transfer a ton of exact information across the image (and that information can't easily be embedded in 2d. to know the reflection, you have to have a 3d map).
- notahacker 4y ago
  Eyeballs appearing to point in different directions and really asymmetric ears are among the common 'tells' of AI generation in a lot of models too. I think the biggest challenge is that unlike facial symmetry, reflected mountains in training sets are often significantly perturbed by ripples and objects in the water. So tones in the water broadly matching the tones of the mountains above but with different texture is "good enough" for the model, and it doesn't catch finer detail mismatches in shapes of edges in the same way it catches finer detail mismatches in the shapes of eyes, which almost invariably have the same texture and similar lighting to their compatriot in images.
dahart 4y ago
Faces aren’t perfectly symmetrical though, while mirror reflections are physics. The degree to which the landscape reflection symmetries are off in the article might be acceptable when generating faces. One possible reason the same approach might not work in both cases could be that DALLE’s training set has more faces than landscapes with still water lakes, it’d have to learn these two symmetry constraints through enough clear examples in both cases, and no counter-examples. I can imagine counter-examples for both faces and landscapes that could potentially inhibit learning symmetry, like faces from an angle or side view, or landscapes with an occluder, but I have no idea how often these occur in DALLE’s training set. Getting this right means that DALLE somehow needs to learn the concept and the symmetry constraint from all the imagery, it has to understand the world a little, not just have pixels to borrow. It’s a hard problem, and at some level surprising how well it’s doing with the landscapes.

deleted 4y ago

[deleted]

coding123 4y ago

It's actually really interesting that the mirror image is that close though. The AI could have seen the water and just saw that there is mountainous shapes, but the AI at least noticed that it's similar to the above one - which tells you how deep these nets go. Perhaps for it to know that it's a nearly perfect mirror image it will need more depth.

tauntz 4y ago

I had to try quite hard to spot issues with the reflections after reading the comments here. Maybe I'm wired differently