4 ms·
"On each fold, the trained classifier was tested on the data of the left-out participant. This procedure was reiterated for all 34 possible ways of leaving out
by asperous 9y ago
"On each fold, the trained classifier was tested on the data of the left-out participant. This procedure was
reiterated for all 34 possible ways of leaving out one participant, yielding 34 classifications whose
averaged accuracies are reported."
Sounds like they overfit their cross validation score and reported that. The data is actually available here though:
http://www.ccbi.cmu.edu/Suicidal-ideation-NATHUMBEH2017/ http://www.ccbi.cmu.edu/Suicidal-ideation-NATHUMBEH2017/
- jjoonathan 9y agoHow does LOOCV overfit the cross validation score?
- mabbo 9y agoThe simplest answer is that every single data point was used in some part of the training, and then later used to give test validation. We have no idea how this result would work on a new data point that has not been used in training. It's bad statistics, bad data science.
- jjoonathan 9y agoMulti-fold cross validation is an established technique. At no point is any data that was used to train a model also used to validate it. > One round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are combined (e.g. averaged) over the rounds to estimate a final predictive model. https://en.wikipedia.org/wiki/Cross-validation_(statistics) https://en.wikipedia.org/wiki/Cross-validation_(statistics) You might object that it's difficult to achieve the level of inter-fold isolation required to make the technique sound, and indeed if you search the comments there's some question as to whether or not there was an information leak in their feature selection process. In that sense calling it "bad statistics, bad data science" might be reasonable, but it's also a powerful technique, so I don't think it's reasonable to dismiss out of hand without being more specific.
- deleted 9y ago[deleted]
- perfmode 9y agoLOOCV can have high variance, but overfitting?
- smhost 9y agoWhat's the smallest sample size anyone has used to get > 90% on the MNIST set?
- asperous 9y agoI was thinking they might have "overfit" by picking hyper-parameters that happened by chance to result in a high CV score but wouldn't perform well in a holdout set. I could be wrong though, that's just my intuition.