3 ms·
Photographic images generated by these systems tend to look like the graffiti portraits you see on fairground attractions. I've done a lot of photorealistic dr
by smokel 2y ago
Photographic images generated by these systems tend to look like the graffiti portraits you see on fairground attractions.
I've done a lot of photorealistic drawings, and the trick to make something look real, is to get the tones exactly right. Misjudge a tone a bit, and the result looks like a mediocre drawing or a painting. In other words, the gradient of skin tones is off, which is ironic, I guess.
I assume that there is a systemic error in (linearly?) interpolating colors (in the wrong color space?) somewhere, which potentially could be easy to fix and lead to improved photorealism. On the other hand, it might be a horrible problem to fix, because it would require accurate radiosity and raytracing to get right.
- jsheard 2y agoI know what you mean, my theory is that it's an emergent property from RLHF tuning penalizing examples of bad/incoherent lighting, which pushes the model towards that kind of vague "lit from everywhere" style which is relatively easy to sell as correct without a proper understanding of light transport. It looks amateurish because that's the same trick an amateur human might use to try to sell photorealism without good lighting fundamentals.
- GaggiX 2y agoThe fact that these models rely heavily on classifier-free guidance has a strong impact on the tones of the image.
- bastawhiz 2y agoExactly. No humans are involved in most of the process.
- TylerE 2y agoIt doesn’t help that RGB is very badly tuned for many skin tones.
- jsheard 2y agoDo these models really operate in RGB space? I would have thought that using a perceptual color space to generate images meant to be perceived by humans would be low hanging fruit.
- TylerE 2y agoAt the very least they ultimately output to RGB. The fleshtone part of the spectrum is quite small.
- smokel 2y agoAs a total cluebie on generative art, I would assume that the neural networks involved use linear weights and ReLU only. If the training data and the output are in RGB pixels, then it would be reasonable to suppose that this introduces some bias. It may not be enough to use a perceptual color space only. The gradients in skin tones, or any other complex texture, are non-linear due to lighting and curvature. Is there someone in the room who does know how things work, and whether this hypothesis is wrong or not?
- bastawhiz 2y agoI have a feeling this is partly due to training data being selected by an "aesthetic score". If weirdly airbrushed skin consistently has a high aesthetic score, that's what the model gets trained on.