7 ms·
I have read a huge swathe of academic literature on classifying and recognizing emotions. And the fact is that humans can't reliably identify emotions from pho
by crazygringo 6y ago
I have read a huge swathe of academic literature on classifying and recognizing emotions.
And the fact is that humans can't reliably identify emotions from photographs, so there's zero chance for AI.
While I don't agree with much of what she writes, the psychologist Lisa Feldman Barrett gives an excellent overview of why at the start of her book "How Emotions are Made".
But a short explanation is that 1) emotions are expressed not just with the face but with the whole body, 2) the expression of emotion is a sequence of changes over time that can't be captured in a single frame, 3) there is extreme overlap in the expression of emotions (e.g. a seemingly grimaced look could either be painful anguish or ecstatic surprise) and 4) while we do have instinctual expressions for emotions (smile/frown/etc.), we learn culturally how and when to hide, show, fake, or otherwise modify these by habit, as well as consciously do so for our immediate purposes.
So we humans obviously do recognize emotions in others, but we do so with a huge variety of cues, we still frequently get it wrong, and we do a far better job when we share the same culture, and also know the individual person well. (Think how some people express anger with aggressive shouting, others by silently withdrawing.)
Most evidence for the idea that emotions can be identified (and cross-culturally) comes from the research of Paul Ekman, but this was limited to 6 "basic" emotions with photographs of highly exaggerated/stylized faces performed by actors. It has no relevance to whether real-life, non-"staged" emotions can be reliably detected.
So any AI products designed to supposedly recognize emotions are necessarily snake oil. I assume they are reliably measuring facial expressions, but facial expressions simply cannot be mapped to actual emotions in any kind of reliable way in the real world.
- bumby 6y ago>the fact is that humans can't reliably identify emotions from photographs As a layman, I had previously thought Paul Ekman's work refuted this claim, but it seems like it's been called into question. (Ekman was reportedly the basis for the main character in the show Lie to Me) https://www.nature.com/articles/d41586-020-00507-5 https://www.nature.com/articles/d41586-020-00507-5
- Nasrudith 6y agoTechnicality people do not always express emotions the same way physically despite what may be the "consensus" on what an expression is believed to be. It can differ culturally for one, let alone neurological conditions or neurodiversity. Human vision isn't strongly reliable - just the current standard. If somebody claims they don't have HIV when their tests come back positive it is clear which one wins out.
- Wistar 6y agoI have certainly been surprised by the question, "Are you mad?" from people who know me very well and have completely misread my emotion. And that's in person.
- mLuby 6y ago> And the fact is that humans can't reliably identify emotions from photographs, so there's zero chance for AI. The second doesn't necessarily follow from the first—there are plenty of things humans are bad at that computers are good at. You are probably right that current products aren't very capable, and may even be snake oil. But also I'd be quite surprised if this isn't possible in the near future.
- Tagbert 6y agoStatic photos are inadequate for interpreting emotions. There is no context and context is critical for understanding emotions. What was the situation? What expressions was the person making in the previous moments. As you said, body language. How the person relates to the viewer. A video clip would be better but still would miss a lot.
- dwighttk 6y agoFWIW and I’m not an AI and this is a much smaller question... But there was a little test that ran around a few years back that was just “Is this smile genuine or not” and I was able to identify 20 out of 20 pictures real and fake smiles. The pictures were gathered by asking subjects to smile and then taking the picture or taking a picture after they responded to a joke, so I guess there is still room for some of the “genuine” smiles to be faked.
- crazygringo 6y agoWell, that's actually relatively easy if you know what you're looking for -- real smiles are known as "Duchenne smiles" and involve contraction of eye muscles. And that could absolutely be done by AI today, relatively easily. The issue is the emotion behind it... you can be quite happy without smiling at all, and people who pose for the camera all day long learn to give a 100% convincing Duchenne smile no matter how they're feeling that day.
- goatcode 6y ago> And the fact is that humans can't reliably identify emotions from photographs, so there's zero chance for AI. I believe there are a lot of examples of AI doing things that humans can't. Typically it's a matter of scale, but sometimes it's a matter of some correlations being beyond the basic capabilities of humans. I could be wrong, but I'm not sure that this is the best metric for whether AI could do something.
- crazygringo 6y agoFirst, you need reliable training data to begin with. This can't be supplied by third-party labeling of faces, so it would need to be done with a dataset of self-reported emotions, but self-reported emotions also have tons of pitfalls well documented in the literature. But second, the correlations of individual facial muscle contractions with emotions has been extensively studied and it's far noisier, inconsistent, or completely devoid of a signal than many people assume. In academic terms there's no such thing as a reliable emotional "signature" to be gleaned from facial muscle activation. So the point is, it appears that the raw data simply isn't there for the AI to detect patterns that humans can't. Detecting emotions requires far more data points outside of facial muscle activation -- such as the ones I listed.
- visarga 6y agoUnsupervised learning is really good today, about 1-2% under supervised learning. You can also cross correlate text, audio and video. If there's signal in the data, then a model can learn it, and you don't need so many labels.
- wizzwizz4 6y agoThe signal is facial expressions. You're missing the “map facial expressions to emotions” step.
- unishark 6y agoYour argument is basically that since something is hard it is impossible?
- tablespoon 6y ago> So any AI products designed to supposedly recognize emotions are necessarily snake oil. I assume they are reliably measuring facial expressions, but facial expressions simply cannot be mapped to actual emotions in any kind of reliable way in the real world. Unfortunately snake oil is a profitable business, and unreliability isn't going to stop people from selling and using these systems. It's going to be like AI-driven captchas [1], the people subjected to these systems are going to have to learn to give the systems the answers they want, even if it's obvious they're incorrect. You'll have students concentrating on expressing an "attentive" face rather than concentrating on the material. [1] Yes, I'm talking about you reCAPTCHA. Highway buzz strips aren't crosswalks.
- lacker 6y agoAny AI products designed to supposedly recognize emotions are necessarily snake oil. You are making far too extreme a judgment here. You also say humans cannot "reliably" identify emotions from photographs. This is just setting the bar far too high. Humans can quite often identify emotions from photographs. It's far better than random chance and you shouldn't call this level of performance "snake oil". You can criticize AI emotion detection as likely to lead to some sort of dystopia, but saying it can't possibly work is just going to lead you to an incorrect analysis of the situation.
- crazygringo 6y ago> This is just setting the bar far too high... It's far better than random chance I don't think it is setting the bar too high, "far better than random" can still be abysmal. You're right that humans are better than random, but there is a wide range of emotions. Assume for the sake of argument there are 10 emotions to identify and evenly distributed in frequency, then random is getting it right 10% of the time. Let's say humans get it right 30% of the time from still photographs of real-life situations (not staged) -- I'm making this specific number up but depending on the type of experiment, it's a reasonable ballpark one. That's still horribly low, wrong more than half the time. I can't think of any situation where that could warrant any kind of responsible decision-making. If we were talking 95% accuracy then there could be value, I'm not asking for 99.99% accuracy. But if reliable detection is something like 30%, I stand by calling this "snake oil" even if it's not just random.
- Fernicia 6y agoIf you agree that AI can reach human levels of accuracy is your point that we shouldn't ever try to assess someone's emotion in decision making, whether it's a person or machine deciding?
- jcelerier 6y ago> is your point that we shouldn't ever try to assess someone's emotion in decision making, whether it's a person or machine deciding? isn't that a pretty sane and basic assumption ?
- ianai 6y agoFurther, I’ve had people over-analyze my every expression to the point of abuse plenty of times. It is entirely possible to wind up with unneeded hostility from trying to infer emotional states. I wish people and machines would Not try to infer too much from this sort of thing. Treat people as ends in themselves.
- pessimizer 6y agoAlso, emotions aren't necessarily a thing as such; they're at least sometimes a retroactive interpretation of physical reactions (sympathetic and parasympathetic nervous system activation.) i.e. you know you got a rush, then by examining context, you attach a (theory-theory) word to that rush. Both AI and photographs are bad at context.
- uyt 6y agoWe seem to have evolved to display emotion because it's a useful communication tool. So much that in text based communication where a lot of emotion is lost, we explicitly re-insert emote-icons to make it clear again. So emotions can be thought as a language (albeit a pretty lossy one). If AI can one day understand spoken/written language, I don't see why it can't also understand the language of emotions.
- bitL 6y agoAI can work on real-time videos (see "ancient" techniques with C3D). A single photograph is noisy, a bunch of photographs close spatiotemporally are much more decisive.
- visarga 6y ago> So any AI products designed to supposedly recognize emotions are necessarily snake oil. Please consult the current state of AI before giving an a priori rebuttal. Emotions can also be detected in text, voice and video, not just in static images. Sentiment detection in text is widely used. The premise of the article is valid, whenever AI systems are being deployed they need to be ethically vetted, but the writing is superficial and 'too emotional'.
- crazygringo 6y agoI'm well aware of the current state of AI, thanks. You, however, may need to familiarize yourself with the current state of emotions research. Sentiment detection in text, for example, is not emotion detection. That's not a rebuttal. As an example of what I mean, you can feed a NYT news article written by a journalist experiencing regret, or boredom, or elation, due to events at home, while writing it -- and it won't pick up a thing. But it may pick up a valid sentiment of outrage, which was intended by the author as the desired effect upon the reader, but which was not once felt as an emotion by the author while writing it. Semantic detection is far easier because the author of text generally tries to communicate a sentiment, consciously, in their text, and text is intended to have fairly unambiguous meaning. Neither is generally true of emotions on the face. And nowhere did I say emotions only exist in static images. Of course they are communicated through voice, video, etc. -- I don't know how anyone could think otherwise.
- treesprite82 6y ago> And nowhere did I say emotions only exist in static images. Of course they are communicated through voice, video, etc. -- I don't know how anyone could think otherwise. Your initial post argued about it being difficult to detect emotion given just a single frame, then concluded that "any AI products designed to supposedly recognize emotions are necessarily snake oil".
- crazygringo 6y agoI thought it was clear from context that the product in question would be evaluating video by using single frames. And I stand by the snake oil comment for the reasons stated, with the state of AI and emotions research today. I'm not saying it can never be done, but that we would need major breakthroughs in both. Sentiment detection continues to be an entirely different area of research.
- vilhelm_s 6y agoI think these objects are maybe true but not relevant. The leading example of the article was about Zooom videos, not single frames, and in a school setting you presumably know what culture the students are from, so I don't think you need it to work cross-culturally. And I don't doubt that people can deliberately hide their emotions, but many settings are not adversarial, so there would be no reason to do so. E.g., if you imagine that a teacher is teaching a lecture to a large audience over zoom and is using a "puzzlement meter" to see if they seem to be getting it, then any audience member who does feel puzzled will only gain from frowning and making the lecturer slow down. In general, I think from my experience talking to people over video chat, some emotional information does get communicated over the video, so in principle computer programs should be able to pick up on it too.
- crazygringo 6y ago> about Zooom videos, not single frames The products in question are almost certainly assessing still frames from video. There's been vanishingly little research on the time component of emotions, even though we know it's hugely important. > you need it to work cross-culturally This would then require separate training models for e.g. individual subcultures within countries as well as detecting which subcultures participants belong to. That is also far beyond anything being done currently. > but many settings are not adversarial, so there would be no reason to do so It has nothing to do with an adversarial setting, people try to hide their emotions constantly. They hide that they're fed up with their boss in front of colleagues, they hide that they're stressed with their spouse at work, they hide that they're worried the project will fail. We are emotionally regulating virtually all the time. > some emotional information does get communicated over the video, so in principle computer programs should be able to pick up on it too In principle, yes, but in practice the emotional content is so dependent upon your cultural and individual mental model of the person that you would need to model their entire psychology. "What does that long pause mean?" The point is that emotional signals are so incredibly complex and vary so much from person to person, that the difficulty of accurately decoding emotions is more akin to AI that can make conceptual inferences and hold a genuinely intelligent conversation, as opposed to mere pattern recognition.
- nl 6y ago> And the fact is that humans can't reliably identify emotions from photographs, so there's zero chance for AI. This is absolutely not the case. I work in adjacent fields, and it not uncommon for machine learning based solutions to reliably outperform any single human. In most cases ML aggregates human judgement. Provided some humans are right more of the time than a random distribution would give (and there is enough data) then a good learning algorithm will find it. Now there are issues with this of course - inappropriate generalisation, inappropriate overfitting etc. And this comment is correct - data outside a single photo will make the system much more reliable. 2 second video sequences should help a lot. But the general principle is valid: an AI/ML should be able to reliably outperform humans.
- deleted 6y ago[deleted]
- mr_toad 6y ago> And the fact is that humans can't reliably identify emotions from photographs, so there's zero chance for AI. You say that like human intelligence is some sort of upper bound on intelligence.