4 ms·

"Machine learning entails training a classifier on a subset of the data and testing the classifier on an independent subset. The crossvalidation procedure itera

by nonbel 9y ago

"Machine learning entails training a classifier on a subset of the data and testing the classifier on an independent subset. The crossvalidation procedure iterates through all possible partitionings (folds) of the data, always keeping the training and test sets separate from each other. The main machine learning here uses a GNB classifier (using pooled variance). [...] The features used by the classifier to characterize a participant consisted of a vector of activation levels for several (discriminating) concepts in a set of (discriminating) brain locations. To determine how many and which concepts were most discriminating between ideators and controls, a reiterative procedure analogous to stepwise regression was used, first finding the single most discriminating concept and then the second most discriminating concept, reiterating until the next step reduced the accuracy. A similar procedure was used to determine the most discriminating locations (clusters)." https://www.nature.com/articles/s41562-017-0234-y https://www.nature.com/articles/s41562-017-0234-y The winner is #3: data leakage leading them to use predictive skill on the training data.

jjoonathan 9y ago
If they included feature generation in the training process and ran it once per fold, it would be OK, but I still haven't found any evidence that they did this and their wording suggests that they did not. Good catch.
- nonbel 9y ago
  >"If they included feature generation in the training process and ran it once per fold, it would be OK" How so? The data used as validation in one fold would be used to determine features in the next...
- nonbel 9y ago
  Anyway that isn't what they did. From the supplements: "To identify the most discriminating concepts, a reiterative procedure analogous to stepwise regression was performed. In the first iteration, the group classification was performed using only one concept at a time, determining which single concept of the 30 resulted in the highest classification accuracy. In the second iteration, the classification was performed using pairs of concepts, namely the single concept that produced the highest accuracy in the first iteration as well as each of the 29 other concepts. All pairs that produced at least as high an accuracy as achieved on the previous iteration, were explored in the third iteration, where triplets of concepts were used, namely the pairs that produced the highest accuracy in the previous iteration, plus each of the remaining 28 concepts. Such stepwise addition of discriminating concepts continued until adding any one of the remaining concepts resulted in a decrease in accuracy. An analogous procedure identified the most discriminating locations." But I still think even in your case they are doing: train: abc; val: d -> score1/ features0 -> features1 train: abd; val: c -> score2/ features1 -> features2 ...etc score2/features1 would all contain info from c, etc.

by nonbel 9y ago

jjoonathan 9y ago

If they included feature generation in the training process and ran it once per fold, it would be OK, but I still haven't found any evidence that they did this and their wording suggests that they did not. Good catch.

nonbel 9y ago
>"If they included feature generation in the training process and ran it once per fold, it would be OK" How so? The data used as validation in one fold would be used to determine features in the next...
nonbel 9y ago
Anyway that isn't what they did. From the supplements: "To identify the most discriminating concepts, a reiterative procedure analogous to stepwise regression was performed. In the first iteration, the group classification was performed using only one concept at a time, determining which single concept of the 30 resulted in the highest classification accuracy. In the second iteration, the classification was performed using pairs of concepts, namely the single concept that produced the highest accuracy in the first iteration as well as each of the 29 other concepts. All pairs that produced at least as high an accuracy as achieved on the previous iteration, were explored in the third iteration, where triplets of concepts were used, namely the pairs that produced the highest accuracy in the previous iteration, plus each of the remaining 28 concepts. Such stepwise addition of discriminating concepts continued until adding any one of the remaining concepts resulted in a decrease in accuracy. An analogous procedure identified the most discriminating locations." But I still think even in your case they are doing: train: abc; val: d -> score1/ features0 -> features1 train: abd; val: c -> score2/ features1 -> features2 ...etc score2/features1 would all contain info from c, etc.