3 ms·
Regarding the opening image: if it can't correctly put the marks on dice, how can it put eyes, nose and mouth correctly on a human face?
by amelius 4y ago
Regarding the opening image: if it can't correctly put the marks on dice, how can it put eyes, nose and mouth correctly on a human face?
- bastawhiz 4y agoPresumably the number of faces in the training set far exceeds the number of dice by more than a few orders of magnitude.
- amelius 4y agoIn one of the other posts I noticed this option: > GFPGAN Face Correction: Automatically correct distorted faces with a built-in GFPGAN option, fixes them in less than half a second So apparently there is still an issue with faces.
- SturgeonsLaw 4y agoYep, it gets faces mostly right, but as they say, the devil's in the details. Eyes in particular don't seem to have clearly delineated concentric circles for irises and pupils, instead they are often rendered as a "swirl". Example image directly from Stable Diffusion: https://i.imgur.com/XSk8fIv.png https://i.imgur.com/XSk8fIv.png And here is that image run through GFPGAN: https://i.imgur.com/I53AGmh.png https://i.imgur.com/I53AGmh.png Interesting to note how specialised GFPGAN is, as some of the other details (flowers, hair) seem to be worse in the processed image. I plan to finish this image by manually blending the best of both pictures.
- userbinator 4y agoIronically, that almost looks like the sort of fantasy image you'd find as GPU box art many years ago.
- eminence32 4y agoIn my experience, Stable Diffusion is also pretty bad about human faces and hands too
- ShamelessC 4y ago> Regarding the opening image: if it can't correctly put the marks on dice, how can it put eyes, nose and mouth correctly on a human face? It helps if you consider it all as effectively advanced compression. Everything the model can do is limited by its architecture, the number of parameters in the model, and the accuracy and size of the training data. The underlying architecture is a transformer (e.g. GPT3) wired to a (denoising) diffusion model. Current flaws with this approach: - Transformers seem to approach a "bag-of-words" model, often ignoring the ordering of the words. Among other things, this means that text-to-image models are very bad at "binding attributes" [0]. This is why "a boy wearing a red shirt and a girl wearing a black jacket" may fail (putting the colors on the wrong items, for instance). - Autoregressive transformers have no means to correct early mistakes. - Training data is captioned images and the captions are likely noisy and under-specified. Every time it sees a face labeled as "face" - it tries to generate a face from the distribution of _all_ faces in the data. The same goes for the dice. If the dice are just labeled "dice", but don't have a description of how they landed - the model has to guess which angle you're referring to. As a sibling comment points out, this is exacerbated by the relative frequencies of examples of the data in the dataset. [0] https://wikiless.org/wiki/Binding_(linguistics) https://wikiless.org/wiki/Binding_(linguistics)
- zenlikethat 4y agoIt can’t. :) Well, I kid a bit. I’ve seen it produce some amazing results, but, generally, it has a hard time with that. Often faces end up looking blurry or having these creepy, dead white eyes. Hands likewise often end up malformed (seven fingers anyone?) and twisty. But, it seems to have a much easier time generating passable faces in close ups with the right key words. Especially if you give it an input image that already has a clear one. It also seems to have an easier time doing faces it already knows like a celebrity, presumably because it’s using a strong existing influence instead of inventing/hallucinating it. Supposedly this is improved in their new 1.5 version which is in beta. The software is so compelling that I suspect this will be improved quite quickly. Also, I think either way workarounds will emerge, either by composing with other networks/software (some UIs have GANs for face correction) or the old fashioned way by photoshopping over the blemishes.
- MattRix 4y agoIt’s worth noting you also get MUCH better faces and hands if you’re willing to run it for more steps (100-150). It takes a lot longer to run it at higher step counts so a lot of people don’t do it.