4 ms·

Computer program can detect depression in blogs and online texts

17 points by mudil 16y ago

Researchers have developed a program that detects depression in text without obvious terms like "depression" or "suicide". In a sample of 200 positively identified texts out of 300,000 which were screened by the program, there was a 78 percent agreement between the program and a panel of psychologists.

mikecane 16y ago
How soon before there's software that can "discover" who is gay, who is bi, who votes this way or that way, whose language is creditworthy, et al? I await the day when all of this is discredited.
- showerst 16y ago
  It's a relatively simple question whether a certain trait can be 'ascertained' by these systems, and to what accuracy, in a given context. There are a number NLP programs that can detect a writers gender from text at a much greater accuracy than raw chance, especially in a limited domain. The problem is that the world is a big, messy domain. There's no reason in principal that other traits wouldn't be 'detectable' (although I doubt sexuality), but the problem is getting a good set of training/test data in context, and what your acceptable threshold for accuracy is. I'd be willing to bet that people whose text messages contain lots of U's and L8R's probably have worse credit than those who text in proper English, but there are much better ways of doing a credit check. If that's only accurate to, say, 100 FICA score points it's probably not worth using it. I wouldn't be too worried about big brother(s) using semantic analysis to read your mind, but don't discredit the fact that language patterns encode certain cultural information that is identifiable, or at least correlated with behavior.
  - mikecane 16y ago
    >>>or at least correlated with behavior. Oh really? Briton Convicted for ‘Menacing’ Tweet Against Robin Hood Airport http://thelede.blogs.nytimes.com/2010/05/10/briton-convicted-for-menacing-tweet-against-robin-hood-airport/ http://thelede.blogs.nytimes.com/2010/05/10/briton-convicted...
  - rada 16y ago
    I tested one of those programs once. I don't remember a 100% but the list of "women" writers that came up for me was something like, Linus Torvalds, Richard Dawkins, Steve Yegge, etc. I think I tested about a dozen male bloggers in total, and most of them came up female.
- tokenadult 16y ago
  How soon before there's software that can "discover" who is gay Do you mean this? http://www.boston.com/bostonglobe/ideas/articles/2009/09/20/project_gaydar_an_mit_experiment_raises_new_questions_about_online_privacy/ http://www.boston.com/bostonglobe/ideas/articles/2009/09/20/... http://www.boston.com/bostonglobe/ideas/brainiac/2009/10/project_gaydar.html http://www.boston.com/bostonglobe/ideas/brainiac/2009/10/pro... (I learned about this on Hacker News back when it was first reported.)
  - mikecane 16y ago
    Yeah, I heard of that at the time but didn't want to go back and edit. The worst thing that can happen is the Context Trap: http://en.wikipedia.org/wiki/On_being_sane_in_insane_places http://en.wikipedia.org/wiki/On_being_sane_in_insane_places 1) Humans want to look for X 2) Design software to look for X 3) Software finds X 4) Humans believe software but software is incorrect 5) X is screwed
- samatman 16y ago
  Google Ads seems to 'know' when I'm depressed, suggested marriage counseling when I was going through a divorce, and has otherwise done a pretty good job of scanning my email and suggesting links to me. This kind of analysis will never be perfect, but then, real humans aren't perfect at it either. I think we can predict that software will get better at this kind of 'guessing' over time. It's not half bad right now.
rada 16y ago
However, so far no efforts have been made to actually correlate the program's judgment with the mental state of the writers. So, it's never been tested? Say what?
- dinedal 16y ago
  According to the article, it has been tested, but only to see if psychologists also agree if the text was written by someone that was depressed. No one actually looked at the author and determined if the author was depressed. In otherwords, the program agrees with psychologists 78% of the time if a sample of text may indicate the author is depressed. We're not close at all to a magical diagnosis machine...
- raganwald 16y ago
  I read the post to mean that the computer program correlated highly with the opinion of a panel of human experts when both are given the writing to study. What hasn't been established is the correlation between the program and the mental state of the authors as measured using more conventional methods.
  - rada 16y ago
    Yes, I saw that too, I am a big fan of reading before commenting :) However, I don't think substituting one human error (that of the programmers) by another human error (that of the psychologists) proves anything. You could say that the second group is more qualified as domain experts but it still doesn't make up for complete lack of real life testing. If it did, you wouldn't have to conduct human trials for drugs - you'd just get a bunch of doctors to say that they think the drug works. The fact that the program purports to detect depression, something that amounts to artificial intelligence, makes me especially skeptical. Not to mention the software author's example of using combinations of the word "black" with terms describing "sleep deprivation". By this definition, anyone who blogs about, say, New York Fashion Week, is depressed.
    - samatman 16y ago
      This problem could be mitigated by testing the panel, rather than testing the internet sources. It's a non-trivial problem to get a representative sample of bloggers to submit to a valid screening for depression. On the other hand, collecting a data set of writings by depressed vs. non-depressed people is easier, such data being (I suspect) readily available. If the panel of experts can correlate between depressed writing and depressed people, then there is validity to their correlations of the writing of unknown people, without having to screen the unknowns separately for depression. I suspect even a panel of non-experts could screen writing into depressed and non-depressed. Humans are pretty good at detecting malaise in others.
chronomex 16y ago
Some time ago, a friend of mine trained a Bayesian spam filter on the Livejournal latest posts page, and bucketed posts into "angsty" and "not angsty". It worked surprisingly well.
- stcredzero 16y ago
  This might be a good way of auto-classifying teenagers into market-segment cohorts. It seems to me that one should be able to recognize subculture as well as socioeconomic level. Hmmm, this could actually be worth a heck of a lot of money!
jgrahamc 16y ago
What's the false positive rate? Looks like the false negative rate is 22%. Given the low rate of depression I'd be interested to know if this is in any way useful as a diagnostic tool. Oh. Actually in take that back. We have no idea what the FN rate is. They haven't tested it at all.
tsiki 16y ago
I'm not sure how this is newsworthy - they're basically just detecting texts with certain words in them. It sounds impressive, sure, but so does credit score when you describe it as an algorithm that can predict if people will get into financial trouble.

17 points by mudil 16y ago

mikecane 16y ago

How soon before there's software that can "discover" who is gay, who is bi, who votes this way or that way, whose language is creditworthy, et al? I await the day when all of this is discredited.

showerst 16y ago
It's a relatively simple question whether a certain trait can be 'ascertained' by these systems, and to what accuracy, in a given context. There are a number NLP programs that can detect a writers gender from text at a much greater accuracy than raw chance, especially in a limited domain. The problem is that the world is a big, messy domain. There's no reason in principal that other traits wouldn't be 'detectable' (although I doubt sexuality), but the problem is getting a good set of training/test data in context, and what your acceptable threshold for accuracy is. I'd be willing to bet that people whose text messages contain lots of U's and L8R's probably have worse credit than those who text in proper English, but there are much better ways of doing a credit check. If that's only accurate to, say, 100 FICA score points it's probably not worth using it. I wouldn't be too worried about big brother(s) using semantic analysis to read your mind, but don't discredit the fact that language patterns encode certain cultural information that is identifiable, or at least correlated with behavior.
- mikecane 16y ago
  >>>or at least correlated with behavior. Oh really? Briton Convicted for ‘Menacing’ Tweet Against Robin Hood Airport http://thelede.blogs.nytimes.com/2010/05/10/briton-convicted-for-menacing-tweet-against-robin-hood-airport/ http://thelede.blogs.nytimes.com/2010/05/10/briton-convicted...
- rada 16y ago
  I tested one of those programs once. I don't remember a 100% but the list of "women" writers that came up for me was something like, Linus Torvalds, Richard Dawkins, Steve Yegge, etc. I think I tested about a dozen male bloggers in total, and most of them came up female.
tokenadult 16y ago
How soon before there's software that can "discover" who is gay Do you mean this? http://www.boston.com/bostonglobe/ideas/articles/2009/09/20/project_gaydar_an_mit_experiment_raises_new_questions_about_online_privacy/ http://www.boston.com/bostonglobe/ideas/articles/2009/09/20/... http://www.boston.com/bostonglobe/ideas/brainiac/2009/10/project_gaydar.html http://www.boston.com/bostonglobe/ideas/brainiac/2009/10/pro... (I learned about this on Hacker News back when it was first reported.)
- mikecane 16y ago
  Yeah, I heard of that at the time but didn't want to go back and edit. The worst thing that can happen is the Context Trap: http://en.wikipedia.org/wiki/On_being_sane_in_insane_places http://en.wikipedia.org/wiki/On_being_sane_in_insane_places 1) Humans want to look for X 2) Design software to look for X 3) Software finds X 4) Humans believe software but software is incorrect 5) X is screwed
samatman 16y ago
Google Ads seems to 'know' when I'm depressed, suggested marriage counseling when I was going through a divorce, and has otherwise done a pretty good job of scanning my email and suggesting links to me. This kind of analysis will never be perfect, but then, real humans aren't perfect at it either. I think we can predict that software will get better at this kind of 'guessing' over time. It's not half bad right now.

rada 16y ago

However, so far no efforts have been made to actually correlate the program's judgment with the mental state of the writers. So, it's never been tested? Say what?

dinedal 16y ago
According to the article, it has been tested, but only to see if psychologists also agree if the text was written by someone that was depressed. No one actually looked at the author and determined if the author was depressed. In otherwords, the program agrees with psychologists 78% of the time if a sample of text may indicate the author is depressed. We're not close at all to a magical diagnosis machine...
raganwald 16y ago
I read the post to mean that the computer program correlated highly with the opinion of a panel of human experts when both are given the writing to study. What hasn't been established is the correlation between the program and the mental state of the authors as measured using more conventional methods.
- rada 16y ago
  Yes, I saw that too, I am a big fan of reading before commenting :) However, I don't think substituting one human error (that of the programmers) by another human error (that of the psychologists) proves anything. You could say that the second group is more qualified as domain experts but it still doesn't make up for complete lack of real life testing. If it did, you wouldn't have to conduct human trials for drugs - you'd just get a bunch of doctors to say that they think the drug works. The fact that the program purports to detect depression, something that amounts to artificial intelligence, makes me especially skeptical. Not to mention the software author's example of using combinations of the word "black" with terms describing "sleep deprivation". By this definition, anyone who blogs about, say, New York Fashion Week, is depressed.
  - samatman 16y ago
    This problem could be mitigated by testing the panel, rather than testing the internet sources. It's a non-trivial problem to get a representative sample of bloggers to submit to a valid screening for depression. On the other hand, collecting a data set of writings by depressed vs. non-depressed people is easier, such data being (I suspect) readily available. If the panel of experts can correlate between depressed writing and depressed people, then there is validity to their correlations of the writing of unknown people, without having to screen the unknowns separately for depression. I suspect even a panel of non-experts could screen writing into depressed and non-depressed. Humans are pretty good at detecting malaise in others.

chronomex 16y ago

Some time ago, a friend of mine trained a Bayesian spam filter on the Livejournal latest posts page, and bucketed posts into "angsty" and "not angsty". It worked surprisingly well.

stcredzero 16y ago
This might be a good way of auto-classifying teenagers into market-segment cohorts. It seems to me that one should be able to recognize subculture as well as socioeconomic level. Hmmm, this could actually be worth a heck of a lot of money!

jgrahamc 16y ago

What's the false positive rate? Looks like the false negative rate is 22%. Given the low rate of depression I'd be interested to know if this is in any way useful as a diagnostic tool. Oh. Actually in take that back. We have no idea what the FN rate is. They haven't tested it at all.

tsiki 16y ago

I'm not sure how this is newsworthy - they're basically just detecting texts with certain words in them. It sounds impressive, sure, but so does credit score when you describe it as an algorithm that can predict if people will get into financial trouble.