9 ms·

Machine learning of neural representations of emotion identifies suicidal youth

103 points by phr4ts 9y ago

chris_wot 9y ago
So what will they do after they detect you are suicidal? Stick you in a psych ward? Yet more attempts at taking away the rights of those going through trauma.
- fao_ 9y ago
  That seems like putting the cart before the horse. Diagnosis tools could mean faster access to treatment. Currently in the UK the waiting list for access to mental health treatment is on the range of two to three years. Transforming "suicidal ideation" from a "vague human-given diagnosis" to "tool-given diagnosis" makes it politically easier to push for that. In any case, that's not going to happen based off a single study with 91% accuracy.
chiefalchemist 9y ago
Slightly off topic but the book "Change your brain change your life" was pretty interesting. Perhaps not as scientific as some would prefer, but none the less thought provoking.
gremlinsinc 9y ago
Maybe they should use this test before gun purchases... I don't think someone suicidal should purchase a gun...hell I don't care if they kill themselves, but lately a lot of suicides were mass suicides...we don't need more of that shit.
- Gibbon1 9y ago
  Problem is people buy guns when they aren't suicidal. Tip: If you own a gun and are feeling suicidal, give it to a trusted person for safekeeping.
  - marcoperaza 9y ago
    It can't hurt to get rid of the gun if you're suicidal. But would it actually make a difference? Suicide rates across countries aren't related to gun availability.
    - Gibbon1 9y ago
      Not having ready means to off yourself when you are suicidal can't hurt?
    - aplummer 9y ago
      Suicide is definitely linked to gun availability. "A study by the Harvard School of Public Health of all 50 U.S. states reveals a powerful link between rates of firearm ownership and suicides. Based on a survey of American households conducted in 2002, HSPH Assistant Professor of Health Policy and Management Matthew Miller, Research Associate Deborah Azrael, and colleagues at the School’s Injury Control Research Center (ICRC), found that in states where guns were prevalent—as in Wyoming, where 63 percent of households reported owning guns—rates of suicide were higher. The inverse was also true: where gun ownership was less common, suicide rates were also lower." https://www.hsph.harvard.edu/news/magazine/guns-and-suicide/ https://www.hsph.harvard.edu/news/magazine/guns-and-suicide/
      - wahern 9y ago
        Gun ownership in the U.S. is strongly correlated with socio-economic status, locality, etc. Everybody points to the Australian example, where suicides declined after the 1996 gun control legislation. But unemployment in Australia peaked in 1995 and declined precipitously afterward until 2009. Given everything we know about suicide rates in other countries, and about changes in suicide rates domestically (e.g. recent increase as gun ownership goes _down_[1]), it would very odd if gun ownership was a root cause of suicide. That said, in a country with a strong gun culture like the U.S., I would totally expect a generational dip in suicides if we substantially removed access to guns. But then I'd expect it to normalize when suicidal individuals became more comfortable with other methods. Just like with mass shootings, there's a strong imitation effect. Take away the model that people imitate and it might be awhile until there's a regression to the mean. Even so, that's still reasonable justification for limiting access to guns--saving tens of thousands of individuals. I'm not sure I'd agree with such a policy prescription because of the insane gun politics, but it's quite defensible from a public health perspective. [1] Number of guns have increased but they're concentrated in fewer households.
- sctb 9y ago
  The guidelines ask us to please not do this kind of thing: > Avoid unrelated controversies and generic tangents. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
nonbel 9y ago
Will it be that 10% of people are suicidal and it always predicts non-suicidal? Will it be that accuracy actually means AUC? Will it be that they are reporting predictive skill on the training data?
- nonbel 9y ago
  "Machine learning entails training a classifier on a subset of the data and testing the classifier on an independent subset. The crossvalidation procedure iterates through all possible partitionings (folds) of the data, always keeping the training and test sets separate from each other. The main machine learning here uses a GNB classifier (using pooled variance). [...] The features used by the classifier to characterize a participant consisted of a vector of activation levels for several (discriminating) concepts in a set of (discriminating) brain locations. To determine how many and which concepts were most discriminating between ideators and controls, a reiterative procedure analogous to stepwise regression was used, first finding the single most discriminating concept and then the second most discriminating concept, reiterating until the next step reduced the accuracy. A similar procedure was used to determine the most discriminating locations (clusters)." https://www.nature.com/articles/s41562-017-0234-y https://www.nature.com/articles/s41562-017-0234-y The winner is #3: data leakage leading them to use predictive skill on the training data.
  - jjoonathan 9y ago
    If they included feature generation in the training process and ran it once per fold, it would be OK, but I still haven't found any evidence that they did this and their wording suggests that they did not. Good catch.
    - nonbel 9y ago
      >"If they included feature generation in the training process and ran it once per fold, it would be OK" How so? The data used as validation in one fold would be used to determine features in the next...
    - nonbel 9y ago
nieve 9y ago
"This study used machine-learning algorithms (Gaussian Naive Bayes) to identify such individuals (17 suicidal ideators versus 17 controls) with high (91%) accuracy, based on their altered functional magnetic resonance imaging neural signatures of death-related and life-related concepts." Anyone with a Nature subscription want to check whether they simply trained their discriminator and then used it on the same data set? There's no mention in the abstract of testing it against a fresh control set and that's not promising. https://www.nature.com/articles/s41562-017-0234-y?error=cookies_not_supported&code=6e85e088-d042-4f52-a19e-320e2cd0bdd0 https://www.nature.com/articles/s41562-017-0234-y?error=cook...
- sigstoat 9y ago
  scihub has it. looks like leave-one-out cross validation? "A Gaussian Naive Bayes (GNB) classifier trained on the data of 33 out of 34 participants predicted the group membership of the remaining participant with a high accuracy of 0.91 (P<0.000001), correctly identifying 15 of the 17 suicidal participants and 16 of the 17 controls"
  - eksabajt 9y ago
    Correct. They also tested leaving out 9 of each group for cross-validation and got 76% accuracy in that case.
- asperous 9y ago
  "On each fold, the trained classifier was tested on the data of the left-out participant. This procedure was reiterated for all 34 possible ways of leaving out one participant, yielding 34 classifications whose averaged accuracies are reported." Sounds like they overfit their cross validation score and reported that. The data is actually available here though: http://www.ccbi.cmu.edu/Suicidal-ideation-NATHUMBEH2017/ http://www.ccbi.cmu.edu/Suicidal-ideation-NATHUMBEH2017/
  - jjoonathan 9y ago
    How does LOOCV overfit the cross validation score?
evolve2017 9y ago
To the moderators, the title would be more accurate with 'fMRI' as opposed to 'MRI'. The latter is typically used to examine structural brain elements, whereas fMRI is thought to correlate with brain activity and, by extension, thought. Confusing the two would lead to the more unusual conclusion that suicidal ideation is associated with abnormal brain connectivity, while the authors are instead focusing on neuronal activity.
- mintplant 9y ago
  Specifically fMRI measures blood flow across the brain (the BOLD response) which is correlated with neuron activity. It has good spatial resolution but poor temporal resolution [0], compared to EEG which gives you good temporal resolution but poor spatial resolution. [0] i.e. you know with precision where in the brain activity occurred, but less precisely when it occurred in time
- topgear25 9y ago
  I didn't even really realize I had these confused until you pointed it out. This makes a lot more sense and helps me understand the results. At first I was confused at how brain structure analysis predicted suicidal tendencies/thoughts.
iregina 9y ago
"Words like death and cruelty differentially activated the left superior medial frontal area and the medial frontal/anterior cingulate in the individuals with suicidal ideation – these are areas associated with self-referential thought." I wonder how they reacted to "alive" and "humane"
deleted 9y ago
[deleted]
loeg 9y ago
Doesn't 91% seem far too low to be useful for the general population? Consider that only 7% of the background population experiences one or more depressive episode per year[0] (edit: okay maybe 8% in youth). Assuming independence and using the higher 8% background rate figure for youth, .91 * .08 = 7.3% of the population will receive a true positive result and (1-.91) * (1-.08) = 8.3% of the population will receive a false positive result. This is "pretty bad" — false positives outweigh the true positives — making the value of a positive result useless. (Consider what happens to people so-diagnosed as suicidal when in fact they are not (false positives). Involuntary psychiatric imprisonment is a terrible thing if it isn't absolutely necessary.) [0]: https://www.healthline.com/health/depression/facts-statistics-infographic https://www.healthline.com/health/depression/facts-statistic...
- hk__2 9y ago
  > I don't have the stats grounding to come up with the proportion of true positives to false positives, but I suspect this would be "pretty bad" — vastly more false positives than true positives IANAStatistician, but let’s consider the system is right 91% of the time and we try to detect those 7% you mentioned. Let’s take 1000 people. 70 people are depressive and 930 aren’t. Out of those, 700.91=63 will be correctly classified as depressive by the system and 9300.91=846 will be correctly classified as non-depressive. That leaves us with 63 positives, 846 negatives, 7 false negatives and 84 false positives. False positives largely outnumber false negatives, but they also outnumber the true positives. (if a statistician read this, please correct me if I’m wrong)
  - loeg 9y ago
    Yeah. I came to largely the same conclusion but used the 8% number for youth, given the current title.
- PeachPlum 9y ago
  Isn't the point of research to advance science one step at a time, not go from "does this look promising" to "yes, it works perfectly 100% of the time" in a single quantum leap.
avip 9y ago
It's the kind of "studies" you call BS on first, then go on to figure out the details. Not a very scientific process for sure, but always produces the correct result. https://www.naturalblaze.com/2017/03/scandal-mri-brain-imaging-completely-unreliable.html https://www.naturalblaze.com/2017/03/scandal-mri-brain-imagi...
- verall 9y ago
  At least post the original article[0] if not the paper[1], rather than some weird alt-health website. [0] https://www.sciencealert.com/a-bug-in-fmri-software-could-invalidate-decades-of-brain-research-scientists-discover https://www.sciencealert.com/a-bug-in-fmri-software-could-in... [1] http://www.pnas.org/content/113/28/7900.abstract http://www.pnas.org/content/113/28/7900.abstract
m3kw9 9y ago
Basically you extract a matrix representation of the active or inactive regions that is classified and have DNN learn it like you would learn images, is that a correct assumption?

103 points by phr4ts 9y ago

So what will they do after they detect you are suicidal? Stick you in a psych ward? Yet more attempts at taking away the rights of those going through trauma.

fao_ 9y ago
That seems like putting the cart before the horse. Diagnosis tools could mean faster access to treatment. Currently in the UK the waiting list for access to mental health treatment is on the range of two to three years. Transforming "suicidal ideation" from a "vague human-given diagnosis" to "tool-given diagnosis" makes it politically easier to push for that. In any case, that's not going to happen based off a single study with 91% accuracy.

chiefalchemist 9y ago

Slightly off topic but the book "Change your brain change your life" was pretty interesting. Perhaps not as scientific as some would prefer, but none the less thought provoking.

gremlinsinc 9y ago

Maybe they should use this test before gun purchases... I don't think someone suicidal should purchase a gun...hell I don't care if they kill themselves, but lately a lot of suicides were mass suicides...we don't need more of that shit.

Gibbon1 9y ago
Problem is people buy guns when they aren't suicidal. Tip: If you own a gun and are feeling suicidal, give it to a trusted person for safekeeping.
- marcoperaza 9y ago
  It can't hurt to get rid of the gun if you're suicidal. But would it actually make a difference? Suicide rates across countries aren't related to gun availability.
  - Gibbon1 9y ago
    Not having ready means to off yourself when you are suicidal can't hurt?
  - aplummer 9y ago
    Suicide is definitely linked to gun availability. "A study by the Harvard School of Public Health of all 50 U.S. states reveals a powerful link between rates of firearm ownership and suicides. Based on a survey of American households conducted in 2002, HSPH Assistant Professor of Health Policy and Management Matthew Miller, Research Associate Deborah Azrael, and colleagues at the School’s Injury Control Research Center (ICRC), found that in states where guns were prevalent—as in Wyoming, where 63 percent of households reported owning guns—rates of suicide were higher. The inverse was also true: where gun ownership was less common, suicide rates were also lower." https://www.hsph.harvard.edu/news/magazine/guns-and-suicide/ https://www.hsph.harvard.edu/news/magazine/guns-and-suicide/
    - wahern 9y ago
      Gun ownership in the U.S. is strongly correlated with socio-economic status, locality, etc. Everybody points to the Australian example, where suicides declined after the 1996 gun control legislation. But unemployment in Australia peaked in 1995 and declined precipitously afterward until 2009. Given everything we know about suicide rates in other countries, and about changes in suicide rates domestically (e.g. recent increase as gun ownership goes _down_[1]), it would very odd if gun ownership was a root cause of suicide. That said, in a country with a strong gun culture like the U.S., I would totally expect a generational dip in suicides if we substantially removed access to guns. But then I'd expect it to normalize when suicidal individuals became more comfortable with other methods. Just like with mass shootings, there's a strong imitation effect. Take away the model that people imitate and it might be awhile until there's a regression to the mean. Even so, that's still reasonable justification for limiting access to guns--saving tens of thousands of individuals. I'm not sure I'd agree with such a policy prescription because of the insane gun politics, but it's quite defensible from a public health perspective. [1] Number of guns have increased but they're concentrated in fewer households.
sctb 9y ago
The guidelines ask us to please not do this kind of thing: > Avoid unrelated controversies and generic tangents. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

nonbel 9y ago

Will it be that 10% of people are suicidal and it always predicts non-suicidal? Will it be that accuracy actually means AUC? Will it be that they are reporting predictive skill on the training data?

nonbel 9y ago
"Machine learning entails training a classifier on a subset of the data and testing the classifier on an independent subset. The crossvalidation procedure iterates through all possible partitionings (folds) of the data, always keeping the training and test sets separate from each other. The main machine learning here uses a GNB classifier (using pooled variance). [...] The features used by the classifier to characterize a participant consisted of a vector of activation levels for several (discriminating) concepts in a set of (discriminating) brain locations. To determine how many and which concepts were most discriminating between ideators and controls, a reiterative procedure analogous to stepwise regression was used, first finding the single most discriminating concept and then the second most discriminating concept, reiterating until the next step reduced the accuracy. A similar procedure was used to determine the most discriminating locations (clusters)." https://www.nature.com/articles/s41562-017-0234-y https://www.nature.com/articles/s41562-017-0234-y The winner is #3: data leakage leading them to use predictive skill on the training data.
- jjoonathan 9y ago
  If they included feature generation in the training process and ran it once per fold, it would be OK, but I still haven't found any evidence that they did this and their wording suggests that they did not. Good catch.
  - nonbel 9y ago
    >"If they included feature generation in the training process and ran it once per fold, it would be OK" How so? The data used as validation in one fold would be used to determine features in the next...
  - nonbel 9y ago

nieve 9y ago

"This study used machine-learning algorithms (Gaussian Naive Bayes) to identify such individuals (17 suicidal ideators versus 17 controls) with high (91%) accuracy, based on their altered functional magnetic resonance imaging neural signatures of death-related and life-related concepts." Anyone with a Nature subscription want to check whether they simply trained their discriminator and then used it on the same data set? There's no mention in the abstract of testing it against a fresh control set and that's not promising. https://www.nature.com/articles/s41562-017-0234-y?error=cookies_not_supported&code=6e85e088-d042-4f52-a19e-320e2cd0bdd0 https://www.nature.com/articles/s41562-017-0234-y?error=cook...

sigstoat 9y ago
scihub has it. looks like leave-one-out cross validation? "A Gaussian Naive Bayes (GNB) classifier trained on the data of 33 out of 34 participants predicted the group membership of the remaining participant with a high accuracy of 0.91 (P<0.000001), correctly identifying 15 of the 17 suicidal participants and 16 of the 17 controls"
- eksabajt 9y ago
  Correct. They also tested leaving out 9 of each group for cross-validation and got 76% accuracy in that case.
asperous 9y ago
"On each fold, the trained classifier was tested on the data of the left-out participant. This procedure was reiterated for all 34 possible ways of leaving out one participant, yielding 34 classifications whose averaged accuracies are reported." Sounds like they overfit their cross validation score and reported that. The data is actually available here though: http://www.ccbi.cmu.edu/Suicidal-ideation-NATHUMBEH2017/ http://www.ccbi.cmu.edu/Suicidal-ideation-NATHUMBEH2017/
- jjoonathan 9y ago
  How does LOOCV overfit the cross validation score?

evolve2017 9y ago

To the moderators, the title would be more accurate with 'fMRI' as opposed to 'MRI'. The latter is typically used to examine structural brain elements, whereas fMRI is thought to correlate with brain activity and, by extension, thought. Confusing the two would lead to the more unusual conclusion that suicidal ideation is associated with abnormal brain connectivity, while the authors are instead focusing on neuronal activity.

mintplant 9y ago
Specifically fMRI measures blood flow across the brain (the BOLD response) which is correlated with neuron activity. It has good spatial resolution but poor temporal resolution [0], compared to EEG which gives you good temporal resolution but poor spatial resolution. [0] i.e. you know with precision where in the brain activity occurred, but less precisely when it occurred in time
topgear25 9y ago
I didn't even really realize I had these confused until you pointed it out. This makes a lot more sense and helps me understand the results. At first I was confused at how brain structure analysis predicted suicidal tendencies/thoughts.

iregina 9y ago

"Words like death and cruelty differentially activated the left superior medial frontal area and the medial frontal/anterior cingulate in the individuals with suicidal ideation – these are areas associated with self-referential thought." I wonder how they reacted to "alive" and "humane"

deleted 9y ago

[deleted]

loeg 9y ago

Doesn't 91% seem far too low to be useful for the general population? Consider that only 7% of the background population experiences one or more depressive episode per year[0] (edit: okay maybe 8% in youth). Assuming independence and using the higher 8% background rate figure for youth, .91 * .08 = 7.3% of the population will receive a true positive result and (1-.91) * (1-.08) = 8.3% of the population will receive a false positive result. This is "pretty bad" — false positives outweigh the true positives — making the value of a positive result useless. (Consider what happens to people so-diagnosed as suicidal when in fact they are not (false positives). Involuntary psychiatric imprisonment is a terrible thing if it isn't absolutely necessary.) [0]: https://www.healthline.com/health/depression/facts-statistics-infographic https://www.healthline.com/health/depression/facts-statistic...

hk__2 9y ago
> I don't have the stats grounding to come up with the proportion of true positives to false positives, but I suspect this would be "pretty bad" — vastly more false positives than true positives IANAStatistician, but let’s consider the system is right 91% of the time and we try to detect those 7% you mentioned. Let’s take 1000 people. 70 people are depressive and 930 aren’t. Out of those, 700.91=63 will be correctly classified as depressive by the system and 9300.91=846 will be correctly classified as non-depressive. That leaves us with 63 positives, 846 negatives, 7 false negatives and 84 false positives. False positives largely outnumber false negatives, but they also outnumber the true positives. (if a statistician read this, please correct me if I’m wrong)
- loeg 9y ago
  Yeah. I came to largely the same conclusion but used the 8% number for youth, given the current title.
PeachPlum 9y ago
Isn't the point of research to advance science one step at a time, not go from "does this look promising" to "yes, it works perfectly 100% of the time" in a single quantum leap.

avip 9y ago

It's the kind of "studies" you call BS on first, then go on to figure out the details. Not a very scientific process for sure, but always produces the correct result. https://www.naturalblaze.com/2017/03/scandal-mri-brain-imaging-completely-unreliable.html https://www.naturalblaze.com/2017/03/scandal-mri-brain-imagi...

verall 9y ago
At least post the original article[0] if not the paper[1], rather than some weird alt-health website. [0] https://www.sciencealert.com/a-bug-in-fmri-software-could-invalidate-decades-of-brain-research-scientists-discover https://www.sciencealert.com/a-bug-in-fmri-software-could-in... [1] http://www.pnas.org/content/113/28/7900.abstract http://www.pnas.org/content/113/28/7900.abstract

m3kw9 9y ago

Basically you extract a matrix representation of the active or inactive regions that is classified and have DNN learn it like you would learn images, is that a correct assumption?