7 ms·

Will it be that 10% of people are suicidal and it always predicts non-suicidal? Will it be that accuracy actually means AUC? Will it be that they are reportin

by nonbel 9y ago

Will it be that 10% of people are suicidal and it always predicts non-suicidal? Will it be that accuracy actually means AUC? Will it be that they are reporting predictive skill on the training data?

nonbel 9y ago
"Machine learning entails training a classifier on a subset of the data and testing the classifier on an independent subset. The crossvalidation procedure iterates through all possible partitionings (folds) of the data, always keeping the training and test sets separate from each other. The main machine learning here uses a GNB classifier (using pooled variance). [...] The features used by the classifier to characterize a participant consisted of a vector of activation levels for several (discriminating) concepts in a set of (discriminating) brain locations. To determine how many and which concepts were most discriminating between ideators and controls, a reiterative procedure analogous to stepwise regression was used, first finding the single most discriminating concept and then the second most discriminating concept, reiterating until the next step reduced the accuracy. A similar procedure was used to determine the most discriminating locations (clusters)." https://www.nature.com/articles/s41562-017-0234-y https://www.nature.com/articles/s41562-017-0234-y The winner is #3: data leakage leading them to use predictive skill on the training data.
- jjoonathan 9y ago
  If they included feature generation in the training process and ran it once per fold, it would be OK, but I still haven't found any evidence that they did this and their wording suggests that they did not. Good catch.
  - nonbel 9y ago
    >"If they included feature generation in the training process and ran it once per fold, it would be OK" How so? The data used as validation in one fold would be used to determine features in the next...
  - nonbel 9y ago
    Anyway that isn't what they did. From the supplements: "To identify the most discriminating concepts, a reiterative procedure analogous to stepwise regression was performed. In the first iteration, the group classification was performed using only one concept at a time, determining which single concept of the 30 resulted in the highest classification accuracy. In the second iteration, the classification was performed using pairs of concepts, namely the single concept that produced the highest accuracy in the first iteration as well as each of the 29 other concepts. All pairs that produced at least as high an accuracy as achieved on the previous iteration, were explored in the third iteration, where triplets of concepts were used, namely the pairs that produced the highest accuracy in the previous iteration, plus each of the remaining 28 concepts. Such stepwise addition of discriminating concepts continued until adding any one of the remaining concepts resulted in a decrease in accuracy. An analogous procedure identified the most discriminating locations." But I still think even in your case they are doing: train: abc; val: d -> score1/ features0 -> features1 train: abd; val: c -> score2/ features1 -> features2 ...etc score2/features1 would all contain info from c, etc.
lottin 9y ago
Not only that, the researchers admit that 80% of suicidal people deny being suicidal. Then, how can they be sure than the ones in the control group are not suicidal?
- matt4077 9y ago
  That wouldn't call the results into question. It would, in fact, strengthen them. That's because the measured difference between the groups would be lower (because the real difference would be lower if the groups are more alike than you think). Say you're testing a drug that's supposed to make people taller. You don't know it yet, but it really does make everyone grow 10cm overnight. You give it to half of your volunteers, and the other half gets placebo. The next day you find that the first group grew by 10cm compared to the control. Now say your grad student messed up and half of the control group also got the real thing instead of placebo. Those also grew by 10cm, making the average in the control group 5cm, and your treatment group's effect is suddenly lower.
jaibot 9y ago
It's training data. There's 17 suicidal and 17 non-suicidal scans, for a total of 34 scans. They trained 34 models, leaving one scan out each time. Of those 34 models, 31 correctly predicted the left-out scan. IANAStatistician, but this seems like a trash result.
- bduerst 9y ago
  How so? Isn't there a 50% chance of getting it right by pure chance, but they got it right 91% of the time instead?
- nonbel 9y ago
  Cross validation is ok if you do it once, but they repeatedly did it and chose the features based on the results. You can't keep adjusting your model/features based on cross validation performance without overfitting to the training data.
  - jjoonathan 9y ago
    How did they adjust the model/features based on CV performance? It looks to me like they did LOOCV.
    - nonbel 9y ago
      Read the second paragraph I quoted above: "The features used by the classifier to characterize a participant consisted of a vector of activation levels for several (discriminating) concepts in a set of (discriminating) brain locations. To determine how many and which concepts were most discriminating between ideators and controls, a reiterative procedure analogous to stepwise regression was used, first finding the single most discriminating concept and then the second most discriminating concept, reiterating until the next step reduced the accuracy. A similar procedure was used to determine the most discriminating locations (clusters)." The features were chosen using the same data as used to assess predictive skill.
      - yorwba 9y ago

by nonbel 9y ago

Will it be that 10% of people are suicidal and it always predicts non-suicidal? Will it be that accuracy actually means AUC? Will it be that they are reporting predictive skill on the training data?

nonbel 9y ago

"Machine learning entails training a classifier on a subset of the data and testing the classifier on an independent subset. The crossvalidation procedure iterates through all possible partitionings (folds) of the data, always keeping the training and test sets separate from each other. The main machine learning here uses a GNB classifier (using pooled variance). [...] The features used by the classifier to characterize a participant consisted of a vector of activation levels for several (discriminating) concepts in a set of (discriminating) brain locations. To determine how many and which concepts were most discriminating between ideators and controls, a reiterative procedure analogous to stepwise regression was used, first finding the single most discriminating concept and then the second most discriminating concept, reiterating until the next step reduced the accuracy. A similar procedure was used to determine the most discriminating locations (clusters)." https://www.nature.com/articles/s41562-017-0234-y https://www.nature.com/articles/s41562-017-0234-y The winner is #3: data leakage leading them to use predictive skill on the training data.

jjoonathan 9y ago
If they included feature generation in the training process and ran it once per fold, it would be OK, but I still haven't found any evidence that they did this and their wording suggests that they did not. Good catch.
- nonbel 9y ago
  >"If they included feature generation in the training process and ran it once per fold, it would be OK" How so? The data used as validation in one fold would be used to determine features in the next...
- nonbel 9y ago
  Anyway that isn't what they did. From the supplements: "To identify the most discriminating concepts, a reiterative procedure analogous to stepwise regression was performed. In the first iteration, the group classification was performed using only one concept at a time, determining which single concept of the 30 resulted in the highest classification accuracy. In the second iteration, the classification was performed using pairs of concepts, namely the single concept that produced the highest accuracy in the first iteration as well as each of the 29 other concepts. All pairs that produced at least as high an accuracy as achieved on the previous iteration, were explored in the third iteration, where triplets of concepts were used, namely the pairs that produced the highest accuracy in the previous iteration, plus each of the remaining 28 concepts. Such stepwise addition of discriminating concepts continued until adding any one of the remaining concepts resulted in a decrease in accuracy. An analogous procedure identified the most discriminating locations." But I still think even in your case they are doing: train: abc; val: d -> score1/ features0 -> features1 train: abd; val: c -> score2/ features1 -> features2 ...etc score2/features1 would all contain info from c, etc.

lottin 9y ago

Not only that, the researchers admit that 80% of suicidal people deny being suicidal. Then, how can they be sure than the ones in the control group are not suicidal?

matt4077 9y ago
That wouldn't call the results into question. It would, in fact, strengthen them. That's because the measured difference between the groups would be lower (because the real difference would be lower if the groups are more alike than you think). Say you're testing a drug that's supposed to make people taller. You don't know it yet, but it really does make everyone grow 10cm overnight. You give it to half of your volunteers, and the other half gets placebo. The next day you find that the first group grew by 10cm compared to the control. Now say your grad student messed up and half of the control group also got the real thing instead of placebo. Those also grew by 10cm, making the average in the control group 5cm, and your treatment group's effect is suddenly lower.

jaibot 9y ago

It's training data. There's 17 suicidal and 17 non-suicidal scans, for a total of 34 scans. They trained 34 models, leaving one scan out each time. Of those 34 models, 31 correctly predicted the left-out scan. IANAStatistician, but this seems like a trash result.

bduerst 9y ago
How so? Isn't there a 50% chance of getting it right by pure chance, but they got it right 91% of the time instead?
nonbel 9y ago
Cross validation is ok if you do it once, but they repeatedly did it and chose the features based on the results. You can't keep adjusting your model/features based on cross validation performance without overfitting to the training data.
- jjoonathan 9y ago
  How did they adjust the model/features based on CV performance? It looks to me like they did LOOCV.
  - nonbel 9y ago
    Read the second paragraph I quoted above: "The features used by the classifier to characterize a participant consisted of a vector of activation levels for several (discriminating) concepts in a set of (discriminating) brain locations. To determine how many and which concepts were most discriminating between ideators and controls, a reiterative procedure analogous to stepwise regression was used, first finding the single most discriminating concept and then the second most discriminating concept, reiterating until the next step reduced the accuracy. A similar procedure was used to determine the most discriminating locations (clusters)." The features were chosen using the same data as used to assess predictive skill.
    - yorwba 9y ago