4 ms·

Hi everyone, So there seems to be a lot of skepticism about whether our algorithm can actually measure emotional impact accurately. For the long answer, I'll r

by tansey 15y ago

Hi everyone, So there seems to be a lot of skepticism about whether our algorithm can actually measure emotional impact accurately. For the long answer, I'll refer you to the about page[1] for EffectCheck. For the short answer: My co-founders are an AI PhD and a Clinical Psychology PhD. They spent three years curating a huge dictionary of words using a methodology similar to the Harvard Psychosocial Dictionary [2], but with the twist that they were focused on lexical impact of words. The dictionary is pretty accurate at measuring both impact and sentiment [3]. For example, we can predict Amazon reviews as being positive or negative, using the stars to validate if we are correct-- blog post on that coming soon. Regarding context of the word usage: Both behavioral studies and fMRI scans have confirmed that context is not as important as one might believe. Our brains process multiple meanings of words in parallel, and the emotions associated with those words linger in our subconscious even after we know the correct context. Similarly, in cases of reviews and comments, people who use hostility-evoking words are often hostile themselves (angry people tend to make others angry) and the same is true of the other five fundamental emotions we measure. Happy to answer more questions. Also happy to analyze any data that you would like to see in order to verify accuracy of the algorithm-- just give me a link or the text. :) [1] http://effectcheck.com/about http://effectcheck.com/about [2] http://www.wjh.harvard.edu/~inquirer/homecat.htm http://www.wjh.harvard.edu/~inquirer/homecat.htm [3] Note that sentiment is correlated to impact only if the writer or speaker is writing without detailed attention to word choice. For example, political speech writers comb over every word to make sure they have the desired impact-- thus, the final text likely has little correlation to the sentiment of the orator.

ChuckFrank 15y ago
This sounds like solid and worthy research. I would be especially interested to see your Amazon words v. stars results. Just off the top of my head, I can see a ton of uses for your research -- More accurate real time political polling / Better product assessments across the vast internet landscape for both designers and brand managers / Policy analysis for new policy proposal and communication evaluations. This has fantastic potential. Please show us the testing and results when you get to it. Thank you for sharing.
jlees 15y ago
You claim that context is not important, but surely in a domain-specific corpus such as HN comments you have to take domain, if not context, into account? For example, words that are generally positive may be only used in a snarky context in hacker circles, whereas words that are generally negative may similarly be used with positive affect (e.g. the word "hacker" itself). Did you customise your lexicon at all for this genre? Also, in my experience, it's not the lexicon of words -- heck, SentiWordNet has existed for long enough -- it's what you do with that input signal that counts. Many "bag of words" approaches are on the shelf now, and very few of them are particularly accurate or clever, but most of them do the job just well enough that they get away with it.
- tansey 15y ago
  ...surely in a domain-specific corpus such as HN comments you have to take domain, if not context, into account? That's a great insight and I believe you are correct. In general, I think the domain does have an impact and I am working on some automatic demographic profiling technology. However, for a first-look, I think our general word dictionary covers an abundant set of terms that do not vary much between domains. For instance, your comment has the following anxiety-evoking words: not, important, but, example, positive, used, hacker, negative, long, bag, but, away Of all those words, only "hacker" would likely need to be adjusted for the HN domain. For a first look, we'll chalk that up to some minor added noise in the data. EffectCheck does include the ability to automatically profile the document type (e.g., "Press Release" or "Motivational Speech") based on a training corpus. For example, most press releases have a high degree of confidence-- based on a corpus of press releases, we normalize the levels so that "Very High" on confidence for a press release requires much higher confidence elicited per-word than it would in a complaint letter.
  - jlees 15y ago
    Is there really much of interest in press releases, or were they just a handy corpus you had around? I started my PhD off looking at affect in scientific papers, and changed tack because there basically is none. Anyway, I guess I'm not clearly understanding your definition of "anxiety" because few of those words evoke anxiety as I would define it -- not? bag? away? I'll try to dig into whatever you've published to deeper understand your system, but I'm not convinced by the standalone word approach yet, though I see what you're trying to do (I think). I'd really want to check out the bigrams/trigrams, even if you don't do full parsing or any intermediate sophistication. Obviously there's an efficiency aspect, so I guess if your results are good enough already and you have customers, no need to make sacrifices. On a tangent, how does your system interpret common HN words like "healthcare", "Obama" and "education", I wonder? Could you have an "intellectual" or "hacker-ish" dimension based on manual categorisation of words into "this is stuff HN should be talking about" vs "this is stuff that belongs on reddit"?
hugh3 15y ago
I can easily believe that you can predict quite well ratings of Amazon reviews by a simple keyword-counting algorithm. Just by counting words like "good", "enjoyed", "fantastic" vs "terrible", "boring", "awful" you're gonna get a very strong correlation in that limited domain. But what tests have you done on your broader methodology? What experiments can you really do to figure out the extent to which the use of the word "nosegay" is correlated with actual depression? Also, as someone else said, where are the error bars? If there really is a correlation between word choice and other metrics, then some simple statistics should give you error bars on your other metrics, right? Oh, one more thing: in the example on your website you say that the sentence: "That joke kills me!" is "subconsciously" aggressive. My question: would your algorithm rate that at exactly the same level of aggression as the sentence: "I'm gonna kill you!"? cuz, y'know, intuitively one seems rather more aggressive than the other.
- rhizome 15y ago
  It seems that the point is to introduce their special-sauce black box, with an argument to authority about its methodology. I think the correlations you ask for are where the problems will lie, in that there is a value judgement that is being hidden. If I can put myself out on a limb here, I'd say that that measurement is going to be fundamentally unscientific.
- tansey 15y ago
  > Also, as someone else said, where are the error bars? If there really is a correlation between word choice and other metrics, then some simple statistics should give you error bars on your other metrics, right? Please see my response to the original question about this. [1] "That joke kills me!" vs. "I'm going to kill you!" (corrected grammar as we do not usually focus on slang) They do not score exactly the same, though they are both very high on anxiety, hostility, and depression. The former has high marks for happiness and compassion due to the word "joke" being used. [1] http://news.ycombinator.com/item?id=2603946 http://news.ycombinator.com/item?id=2603946
PaulHoule 15y ago
I know this stuff works. Some friends of mine from Cornell were able to train an SVM to predict people's moods on Livejournal.
jarrett 15y ago
Check out the results for their own webpage. Click "Try EffectCheck on this page!" here: http://effectcheck.com/tour http://effectcheck.com/tour I'm a little skeptical. It rates "anxiety" as "medium-high." Yet I don't feel the slightest bit anxious reading that page. Nor does the author come off as anxious. It all feels very positive to me. (I'm counting the text alone, not the graphic design.) It rates "compassion" as "very high." I wasn't able to find a description of what exactly the compassion measurement means in EffectCheck. But I'll go out on a limb and assume a high compassion rating means a text will either make you feel like the author is compassionate, or make you feel compassionate yourself. I don't feel either for that page. Not that it's hostile--I just don't feel anything on that axis when I read the page. It's entirely possible that I've misunderstood how EffectCheck's ratings should be interpreted. It's also possible that I'm missing the point by comparing how I think I feel against EffectCheck's results. Maybe we're unreliable observers of how a text emotionally affects us, and EffectCheck more accurately predicts what's really going on deep in our brains. In which case, forgive my unwarranted naysaying.
- tansey 15y ago
  >It's also possible that I'm missing the point by comparing how I think I feel against EffectCheck's results. Maybe we're unreliable observers of how a text emotionally affects us, and EffectCheck more accurately predicts what's really going on deep in our brains. In which case, forgive my unwarranted naysaying. Bingo. Our algorithm focuses on the lingering, subliminal emotions that you aren't necessarily aware you're feeling. Though not related to the emotions we measure, one commonly-referenced study shows that if you're walking down the street and you see a big sign with the word "ELDERLY" written on it, you'll actually walk slower without realizing it [1]. Lots of words have these subconscious impacts and you simply cannot accurately poll your own brain to determine how you feel. [1] See "Stumbling on Happiness" for a great overview of this study: http://www.amazon.com/gp/product/1400077427/ref=as_li_ss_tl?ie=UTF8&tag=nashc-20&linkCode=as2&camp=217145&creative=399349&creativeASIN=1400077427 http://www.amazon.com/gp/product/1400077427/ref=as_li_ss_tl?...
  - scott_s 15y ago
    How can we falsify your claims? That is, what is a test we can perform that if it went a certain way, would show that your claims are false?

by tansey 15y ago

ChuckFrank 15y ago

This sounds like solid and worthy research. I would be especially interested to see your Amazon words v. stars results. Just off the top of my head, I can see a ton of uses for your research -- More accurate real time political polling / Better product assessments across the vast internet landscape for both designers and brand managers / Policy analysis for new policy proposal and communication evaluations. This has fantastic potential. Please show us the testing and results when you get to it. Thank you for sharing.

jlees 15y ago

You claim that context is not important, but surely in a domain-specific corpus such as HN comments you have to take domain, if not context, into account? For example, words that are generally positive may be only used in a snarky context in hacker circles, whereas words that are generally negative may similarly be used with positive affect (e.g. the word "hacker" itself). Did you customise your lexicon at all for this genre? Also, in my experience, it's not the lexicon of words -- heck, SentiWordNet has existed for long enough -- it's what you do with that input signal that counts. Many "bag of words" approaches are on the shelf now, and very few of them are particularly accurate or clever, but most of them do the job just well enough that they get away with it.

tansey 15y ago
...surely in a domain-specific corpus such as HN comments you have to take domain, if not context, into account? That's a great insight and I believe you are correct. In general, I think the domain does have an impact and I am working on some automatic demographic profiling technology. However, for a first-look, I think our general word dictionary covers an abundant set of terms that do not vary much between domains. For instance, your comment has the following anxiety-evoking words: not, important, but, example, positive, used, hacker, negative, long, bag, but, away Of all those words, only "hacker" would likely need to be adjusted for the HN domain. For a first look, we'll chalk that up to some minor added noise in the data. EffectCheck does include the ability to automatically profile the document type (e.g., "Press Release" or "Motivational Speech") based on a training corpus. For example, most press releases have a high degree of confidence-- based on a corpus of press releases, we normalize the levels so that "Very High" on confidence for a press release requires much higher confidence elicited per-word than it would in a complaint letter.
- jlees 15y ago
  Is there really much of interest in press releases, or were they just a handy corpus you had around? I started my PhD off looking at affect in scientific papers, and changed tack because there basically is none. Anyway, I guess I'm not clearly understanding your definition of "anxiety" because few of those words evoke anxiety as I would define it -- not? bag? away? I'll try to dig into whatever you've published to deeper understand your system, but I'm not convinced by the standalone word approach yet, though I see what you're trying to do (I think). I'd really want to check out the bigrams/trigrams, even if you don't do full parsing or any intermediate sophistication. Obviously there's an efficiency aspect, so I guess if your results are good enough already and you have customers, no need to make sacrifices. On a tangent, how does your system interpret common HN words like "healthcare", "Obama" and "education", I wonder? Could you have an "intellectual" or "hacker-ish" dimension based on manual categorisation of words into "this is stuff HN should be talking about" vs "this is stuff that belongs on reddit"?

hugh3 15y ago

I can easily believe that you can predict quite well ratings of Amazon reviews by a simple keyword-counting algorithm. Just by counting words like "good", "enjoyed", "fantastic" vs "terrible", "boring", "awful" you're gonna get a very strong correlation in that limited domain. But what tests have you done on your broader methodology? What experiments can you really do to figure out the extent to which the use of the word "nosegay" is correlated with actual depression? Also, as someone else said, where are the error bars? If there really is a correlation between word choice and other metrics, then some simple statistics should give you error bars on your other metrics, right? Oh, one more thing: in the example on your website you say that the sentence: "That joke kills me!" is "subconsciously" aggressive. My question: would your algorithm rate that at exactly the same level of aggression as the sentence: "I'm gonna kill you!"? cuz, y'know, intuitively one seems rather more aggressive than the other.

rhizome 15y ago
It seems that the point is to introduce their special-sauce black box, with an argument to authority about its methodology. I think the correlations you ask for are where the problems will lie, in that there is a value judgement that is being hidden. If I can put myself out on a limb here, I'd say that that measurement is going to be fundamentally unscientific.
tansey 15y ago
> Also, as someone else said, where are the error bars? If there really is a correlation between word choice and other metrics, then some simple statistics should give you error bars on your other metrics, right? Please see my response to the original question about this. [1] "That joke kills me!" vs. "I'm going to kill you!" (corrected grammar as we do not usually focus on slang) They do not score exactly the same, though they are both very high on anxiety, hostility, and depression. The former has high marks for happiness and compassion due to the word "joke" being used. [1] http://news.ycombinator.com/item?id=2603946 http://news.ycombinator.com/item?id=2603946

PaulHoule 15y ago

I know this stuff works. Some friends of mine from Cornell were able to train an SVM to predict people's moods on Livejournal.

jarrett 15y ago

Check out the results for their own webpage. Click "Try EffectCheck on this page!" here: http://effectcheck.com/tour http://effectcheck.com/tour I'm a little skeptical. It rates "anxiety" as "medium-high." Yet I don't feel the slightest bit anxious reading that page. Nor does the author come off as anxious. It all feels very positive to me. (I'm counting the text alone, not the graphic design.) It rates "compassion" as "very high." I wasn't able to find a description of what exactly the compassion measurement means in EffectCheck. But I'll go out on a limb and assume a high compassion rating means a text will either make you feel like the author is compassionate, or make you feel compassionate yourself. I don't feel either for that page. Not that it's hostile--I just don't feel anything on that axis when I read the page. It's entirely possible that I've misunderstood how EffectCheck's ratings should be interpreted. It's also possible that I'm missing the point by comparing how I think I feel against EffectCheck's results. Maybe we're unreliable observers of how a text emotionally affects us, and EffectCheck more accurately predicts what's really going on deep in our brains. In which case, forgive my unwarranted naysaying.

tansey 15y ago
>It's also possible that I'm missing the point by comparing how I think I feel against EffectCheck's results. Maybe we're unreliable observers of how a text emotionally affects us, and EffectCheck more accurately predicts what's really going on deep in our brains. In which case, forgive my unwarranted naysaying. Bingo. Our algorithm focuses on the lingering, subliminal emotions that you aren't necessarily aware you're feeling. Though not related to the emotions we measure, one commonly-referenced study shows that if you're walking down the street and you see a big sign with the word "ELDERLY" written on it, you'll actually walk slower without realizing it [1]. Lots of words have these subconscious impacts and you simply cannot accurately poll your own brain to determine how you feel. [1] See "Stumbling on Happiness" for a great overview of this study: http://www.amazon.com/gp/product/1400077427/ref=as_li_ss_tl?ie=UTF8&tag=nashc-20&linkCode=as2&camp=217145&creative=399349&creativeASIN=1400077427 http://www.amazon.com/gp/product/1400077427/ref=as_li_ss_tl?...
- scott_s 15y ago
  How can we falsify your claims? That is, what is a test we can perform that if it went a certain way, would show that your claims are false?