8 ms·
The uncanny failures of A.I.-generated hands
- wmf 4y agoBad timing, since there's a rumor that Midjourney v5 has mostly fixed the hand and teeth problems.
- umeshunni 4y agoGood timing - since they get the clicks for this article and then all the clicks for another article about how it's fixed in a few weeks.
- YurgenJurgensen 4y agoAll the examples of supposedly 'fixed' hands I've seen still had problems, or relied on poses which hid most of the fingers. Even if the obvious 'can't count to five' issue gets fixed, it still has problems correctly rendering knuckles and fingernails and getting good proportions. And people don't even attempt difficult scenes with hands like holding small objects or performing specific actions, presumably because even the most wide-eyed AI-optimist can see how much they don't work.
- jsjohnst 4y ago> And people don't even attempt difficult scenes with hands like holding small objects or performing specific actions They definitely do and with pretty decent (albeit certainly far from perfect) results. I feel you have a bit of cognitive bias based on your sources. A few hundred images of hands well tagged used as fine tuning into an existing SD model produces a dramatic improvement, based on personal experience, including grasping objects. Are they photograph level reproductions, certainly not, but easily qualify as skilled artistic renderings. The part I’ve been experimenting lately on is getting fingernails and skin texture to both be realistic at the same time (I can get one or other, but not both so far).
- YurgenJurgensen 4y agoPeople keep saying things that these models can do, but they're pretty reluctant to actually show any results. Perhaps they're not as good as you claim.
- jsjohnst 4y agoI could show hundreds, but would you trust me or just think they were cherry picked good cases? Skeptical people will be skeptical, news at 11! ;) If you really want to know, you could just try it yourself. Setup WebUI, download some CAI or HF SD models that have been fine tuned with hands, learn about prompting, and then see that the field has dramatically improved by just not using stock settings. Zero to hand model hero in around an hour!
- earthboundkid 4y agoStable diffusion solved being cross eyed by having everyone wear a space helmet. Done, ship it.
- kzrdude 4y agoIs it extra layers and models? They already have some kind of "face fixup" don't they?
- ModernMech 4y agoIt's not just humans, I've had trouble generating appendages for other creatures and also machines like robots. Dragons with missing wings and seven legs, robots with feet that have feet that have feet. Stuff like that. So far the best technique I've found is to fix in retouch photos, or to hide their hands by e.g. prompting characters to have their hands behind their backs.
- zurtri 4y agoAgreed. I have found that generated horses never look right. The legs are routinely wrong. And the horses look more like a cartoon representation than an actual horse.
- wendyshu 4y agoInteresting because hands often look weird in dreams. A trick to know if you're lucid dreaming is to look at your hands.
- felipemnoa 4y agoMaybe only in your dreams. I've never noticed that myself. My dream generator is able to render hands just fine.
- LeoPanthera 4y agoLight switches are a reliable indicator that I am in a lucid dream. Switching them never turns on or off a light. They often cause a small explosion, weirdly. Also, if there is text, it will never read the same twice if you look away from it.
- deleted 4y ago[deleted]
- kragen 4y agotext is another terrible problem for these diffusion models
- tap-snap-or-nap 4y agoI had more issue with gravity than with hands.
- feintruled 4y agoHeard this in a few places. Can't say I've noticed myself (though I never lucid dream). I've even seen it put forward as evidence these models are like the brain as they have similar 'failure modes' - anyone know if there is any truth to that?
- mcaledonensis 4y agoTwo hands have ~60 degrees of freedom (4 in each finger, 4 for extension/flexion/abduction and a few for a pose between the hands). You can speak with your hands! Or perform (shadowgraphy, puppeteering, magic). In computer graphics rendering hands and hands gestures was considered very challenging for a long time (This was, for example, a PhD thesis of one of the Pixar co-founders, back when computer graphics was just starting). And it is still very challenging to render a close-up of a hand performing a natural looking gesture.
- prox 4y agoAs a human drawing hands is a difficult task. Look how many beginners try to avoid exactly the problem you see in SD. You need to break it down in a way so it becomes easier to render. Took me ages.
- atleastoptimal 4y ago> The strange contortions of A.I. hands make me feel a sense of anticipatory nostalgia, for a future when the technology inevitably improves and we will look back on such flaws as a kitschy relic of the “early A.I.” era, the way grainy digital-camera photos are redolent of the two-thousands. This will be true for most AI "failures". People love to harp on some minor issue some new generative AI has in edge cases and act like it's some insurmountable obstacle that will keep humans superior forever: "haha look at the AI try to make a hand what an idiot" As humans become more and more irrelevant, we will cling to these minor failures more and more to preserve our egos
- swatcoder 4y agoThere are cheap optimists and cheap pessimists. Both amount to trolls and don’t particularly need to be fed or heeded. The truth is we don’t know how far today’s breakthroughs in AI will take us. There’s quite a lot of headroom to explore, but history has shown that there are always some brick walls as well. The rich discussions can be good, but the cheap flamewars are exhausting.
- rob74 4y ago> The truth is we don’t know how far today’s breakthroughs in AI will take us That's the central question. So far, AI has proven great at ingesting huge amounts of "prior art" and producing variations on the themes it has learnt (more or less) according to the prompts you give it, but it consistently fails to grasp the underlying rules (gravity, perspective, anatomy etc.) that are supposed to govern the images it generates. So it's better for "surrealist" art than for images that are supposed to be realistic...
- john61 4y agoThe surrealistic manifesto by Breton excludes AI, since it has no thought: "Psychic automatism in its pure state, by which one proposes to express—verbally, by means of the written word, or in any other manner—the actual functioning of thought. Dictated by thought, in the absence of any control exercised by reason, exempt from any aesthetic or moral concern."
- aaron695 4y ago[dead]
- mediaman 4y agoI've been curious about the hand problem. It seems like it’s caused by the massive number of permutations possible with hands: all the joints, combined with all the angles at which they can be seen. I’ve wondered if this could be solved by creating a massive permutation of hand images using rigged 3D hand models (using, for example, Blender, or coupling with Unreal), and programmatically putting them in all possible combinations and angles possible from the rigging and then rendering millions of images of these combinations. Then the image models could learn from that artificially created dataset. Anyone with actual knowledge know how people are trying to tackle this?
- Dalewyn 4y agoIt's really very simple: 1. Humans are terrible at drawing hands and feet (ask any illustrator, amateur or professional). 2. Advanced Interpolation software reads and generates images based off of both real-life photos and illustrations of hands drawn by humans, depending on what materials are fed to it. 3. The images generated carry forth the failings of humanity.
- aetherson 4y agoAI drawn hands are consistently terrible in different ways than bad human artists draw hands badly.
- IIAOPSW 4y ago
- thequadehunter 4y agoUsing Dall-e 2 for all your example images is a great self report that you don't keep up much with AI.
- mattgreenrocks 4y agoThat’d be me. I don’t really care to “keep up” with things. What should I be using instead?
- arthurcolle 4y agoStable Diffusion but it also has artifacts as discussed in the post
- thequadehunter 4y agoStable diffusion finetunes and model merges. You can find most of the newer ones at CivitAI. https://civitai.com/ https://civitai.com/ SD still has the hand problem, but a lot of the newer checkpoints are getting pretty good at hands. As far as keeping up with things, the best way to do this is actually to be in AI related discords. I'm not sure why, but people involved in AI are terrible about having a single source of info...or even just info in general. I would start by joining the LAION discord then finding others from there. Another great way to learn about what's going on in AI is actually through threads on 4chan's /g/ board. If you're willing to look past the fact that it's 4chan you will be the first to know about lots of neat stuff. Btw, I'm not a researcher. I'm just talking about ways to get a quick scoop on what the current AI meta is and how far it can be pushed. edit: I just realized you don't care to keep up...lol. Oops. I'm still leaving this up because I think some people will find it useful.
- erhan24 4y agoYou can position the body and fingers and way you want with ControlNet.
- andsoitis 4y ago> Using Dall-e 2 for all your example images is a great self report that you don't keep up much with AI. I think the overarching story (subtitle of the article) is what is the message: "machines can grasp small patterns but not the unifying whole." Amateur human illustrators might also struggle to draw accurate representations of hands, but, unlike machines, can critically evaluate and KNOW whether they got it right or not...
- spaced-out 4y agoIt is possible to generate good quality hands with Stable Diffusion with ControlNet. Here's a video talking about generating different hand positions from models created in Blender. There's also discussions about this on r/StableDiffusion. https://www.youtube.com/watch?v=ptEZQrKgHAg https://www.youtube.com/watch?v=ptEZQrKgHAg Now from personal experience, it's not as easy as it seems in the video. It still takes a lot of tweaking and many iterations don't come out well, but hands are definitely not an insurmountable problem.
- prox 4y agoDepth controlnets have been the most helpful sofar, they give space directions the best and untangles the good from the bad imo,
- loufe 4y agoI mentioned it in another response as well, but here's the original implementation repo for anyone interested: https://github.com/lllyasviel/ControlNet https://github.com/lllyasviel/ControlNet
- skhr0680 4y agoThe way for AI to make it realistic is to build a human from the bones to the muscles to blood to skin.
- leroy-is-here 4y agoWhere does the training data come from? Humans have traditionally sucked at drawing hands.
- yosefk 4y agoYou mean humans traditionally drew hands with 7 fingers and other such? I have a hard time remembering a single human drawing "sucking" in the way AI-generated hands do.
- kragen 4y ago3 fingers is pretty common because it's easier to get a 3-fingered hand to look okay but yeah humans rarely draw 7-fingered hands or smiles with four rows of 40 teeth
- jsjohnst 4y ago> but yeah humans rarely draw 7-fingered hands Guess you haven’t seen hand drawings done by young children?
- bluefirebrand 4y agoSurely you aren't suggesting that many drawings by young children are in the AI training models and that's why it sucks at drawing hands?
- jsjohnst 4y agoNope, not that at all. I was replying directly to the quoted sentence of GP’s post with a clear and direct rebuttal against it.
- kragen 4y agoi was trying to remember if young children often did this, but came to the conclusion that they only do it rarely
- dieselgate 4y agoDidn’t want to look at the article because I get real intense discomfort when viewing AI generated faces. The hands aren’t as bad but still pretty uncomfortable. It’s almost cubism in a way but still too uncanny valley for me
- kragen 4y agothis worked for me: links2 https://www.newyorker.com/culture/rabbit-holes/the-uncanny-failures-of-ai-generated-hands (in this case i was more interested in not getting paywalled) i was still able to load the images by typing 'i' to launch an external viewer
- bckr 4y agoThanks, I get extremely uncomfortable when looking at the hands. Faces for some reason aren't as bad. But the discomfort is similar to that generated by those older "deep dream" images where everything is a psychedelic octopus dog nightmare
- jononomo 4y agoI wonder if satisfactory AI hands will be like self-driving cars: always five years away.
- GaggiX 4y agoThey are already here, several custom SD models are really good at it.
- GaggiX 4y agoThis article seems a few months late, it is trivial now to get correct hands using custom stable diffusion model or make the slightest commitment to do inpainting if there are errors.
- Groxx 4y agoThey fall apart with just about anything that I'd label "specifically" repetitive (instead of infinitely, generally, or chaotically, e.g. like fences, tree branches, or a pebble riverbed). Stuff where there is a narrow range of correct repetitions, and going outside that is immediately obvious. Hands. Chopsticks. Teeth. Sport balls. Etc. Kinda feels like it gets into loops, but it can't count so it can't tell when to stop.
- yarg 4y agoI think that the problem is that it primarily learns bottom up features from the high frequency domain. The invariants that apply top down structures from the low frequency domain aren't as readily recognised or understood.
- kzrdude 4y agoThat sounds intuitively similar to language models generating syntactically correct sentences, but they don't even fact check the result. Classic chatgpt: "The word contains R because it's spelled L-E-M-O-N"
- earthboundkid 4y agoChatGPT told me that the answer to a riddle was “man” because a crawling baby has three legs and a person with a cane has four legs.
- thom 4y agoTried to get an explanation for a crossword clue today: The answer "gnocchi" is derived from the clue "spouted pan I used for dumplings" using a type of wordplay known as anagram. An anagram is a type of wordplay where the letters of one word or phrase are rearranged to form another word or phrase. In this case, the clue contains an anagram indicator, "spouted," which suggests that the letters of some words need to be rearranged. The words "pan I used" can be rearranged to form the word "nog," and "for dumplings" suggests that we need to add another letter to complete the solution. The letter "c" is a suitable choice here, as it is a common letter in the word "dumplings," and it fits the remaining letters to form the word "gnocchi," which is a type of Italian dumpling made from potatoes. Therefore, "spouted pan I used for dumplings" is a cryptic crossword clue that requires you to rearrange the letters of "pan I used" and add a letter "c" to get the answer "gnocchi."
- irusensei 4y agoIdea: do crimes wearing some toy that makes you look like you have extra fingers. In court point out that evidences are AI generated.
- sam_bristow 4y agoI was thinking of going and editing extra fingers and fucked up teeth into someone's childhood photos.
- ZunarJ5 4y agoSticky Six Fingered Discount has more of a ring to it, I have to admit.
- Jack000 4y agoAn unconditional diffusion model is trying to solve a huge problem - storing the set of all images that are meaningful to humans. I think incorrect details in hands/faces are mostly due to limited model capacity. From the imagen paper we see that details like fingers and text spontaneously get better at around 20b parameters.
- rob74 4y ago> machines can grasp small patterns but not the unifying whole that's actually true for all the art generated by (the current generation of) AI. With some things (like hands) it's more obvious than with others, but the closer you look, the more "wrong" things you notice...
- earthboundkid 4y agoWhen you see the picture as a thumbnail it looks okay. When you look at the details of a full sized picture, it’s a fractal of wrong.
- chaosbolt 4y agoAI pics used to look like things you'd see in a dream. Hands are not different, in fact one of the best ways to find if you're dreaming or not is to try and count your fingers, in a dream your hand would look exactly like a hand generated by AI. I remember reading this on some lucid dreaming subreddit and tried it, you need to get into the habbit of doing it awake so you also get the reflex to do in dream.
- carlmr 4y agoIs that why potheads look at their hands all the time?
- hypertele-Xii 4y agoAll humans look at their hands all the time. It's called hand-eye coordination. Potheads merely consciously realize they're doing it, and are probably more mesmerized at the constant, subconscious looking at hands we all do, than the hands themselves. Having said that, our hands are perhaps the most crucial instrument for manipulating reality, so they are quite amazing.
- mrob 4y agoHand-eye coordination doesn't involve looking at the hands. Hand-eye coordination involves looking at the object you want to interact with, e.g. a button you're going to click, or a ball you're going to catch. You don't look at the mouse or your hands.
- mft_ 4y ago1) Do they? Having spent lots of time around stoned people, I don't remember this. 2) Why the need for a relatively pejorative term, "potheads"?
- loloquwowndueo 4y agoMaybe you were also stoned and thus don’t remember doing it :)
- 0xbadc0de5 4y agoAnyone who knows an artist or has attempted this themselves will immediately confirm that hands and feet are often the most challenging anatomy to get right. I view this result as further evidence that these algorithms are approaching the target.
- mightyham 4y agohttp://web.archive.org/web/20230314012827/https://www.newyorker.com/culture/rabbit-holes/the-uncanny-failures-of-ai-generated-hands http://web.archive.org/web/20230314012827/https://www.newyor...
- erhan24 4y agoZero mentions of ControlNet. Also there is now https://www.reddit.com/r/StableDiffusion/comments/11pyiro/new_feature_zoom_enhance_for_the_a111_webui https://www.reddit.com/r/StableDiffusion/comments/11pyiro/ne...
- loufe 4y agoGithub link to actual source project for anyone curious: >https://github.com/lllyasviel/ControlNet https://github.com/lllyasviel/ControlNet
- andromaton 4y agoFor the old-schoolers: As a kid I watched the hands to see if the TV character was an alien (The Invaders)
- deleted 4y ago[deleted]
- ar9av 4y agoThere's a model called Ziepher's F111 that greatly improves human anatomy. You can merge F111 with other models in automatic1111, something like 80% SD1.5 and 20% F1111 greatly improves human anatomy, including hands.