5 ms·
"This study used machine-learning algorithms (Gaussian Naive Bayes) to identify such individuals (17 suicidal ideators versus 17 controls) with high (91%) accur
by nieve 9y ago
"This study used machine-learning algorithms (Gaussian Naive Bayes) to identify such individuals (17 suicidal ideators versus 17 controls) with high (91%) accuracy, based on their altered functional magnetic resonance imaging neural signatures of death-related and life-related concepts."
Anyone with a Nature subscription want to check whether they simply trained their discriminator and then used it on the same data set? There's no mention in the abstract of testing it against a fresh control set and that's not promising.
https://www.nature.com/articles/s41562-017-0234-y?error=cookies_not_supported&code=6e85e088-d042-4f52-a19e-320e2cd0bdd0 https://www.nature.com/articles/s41562-017-0234-y?error=cook...
- sigstoat 9y agoscihub has it. looks like leave-one-out cross validation? "A Gaussian Naive Bayes (GNB) classifier trained on the data of 33 out of 34 participants predicted the group membership of the remaining participant with a high accuracy of 0.91 (P<0.000001), correctly identifying 15 of the 17 suicidal participants and 16 of the 17 controls"
- eksabajt 9y agoCorrect. They also tested leaving out 9 of each group for cross-validation and got 76% accuracy in that case.
- asperous 9y ago"On each fold, the trained classifier was tested on the data of the left-out participant. This procedure was reiterated for all 34 possible ways of leaving out one participant, yielding 34 classifications whose averaged accuracies are reported." Sounds like they overfit their cross validation score and reported that. The data is actually available here though: http://www.ccbi.cmu.edu/Suicidal-ideation-NATHUMBEH2017/ http://www.ccbi.cmu.edu/Suicidal-ideation-NATHUMBEH2017/
- jjoonathan 9y agoHow does LOOCV overfit the cross validation score?
- mabbo 9y agoThe simplest answer is that every single data point was used in some part of the training, and then later used to give test validation. We have no idea how this result would work on a new data point that has not been used in training. It's bad statistics, bad data science.
- jjoonathan 9y agoMulti-fold cross validation is an established technique. At no point is any data that was used to train a model also used to validate it. > One round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are combined (e.g. averaged) over the rounds to estimate a final predictive model. https://en.wikipedia.org/wiki/Cross-validation_(statistics) https://en.wikipedia.org/wiki/Cross-validation_(statistics) You might object that it's difficult to achieve the level of inter-fold isolation required to make the technique sound, and indeed if you search the comments there's some question as to whether or not there was an information leak in their feature selection process. In that sense calling it "bad statistics, bad data science" might be reasonable, but it's also a powerful technique, so I don't think it's reasonable to dismiss out of hand without being more specific.
- deleted 9y ago[deleted]
- perfmode 9y agoLOOCV can have high variance, but overfitting?
- smhost 9y agoWhat's the smallest sample size anyone has used to get > 90% on the MNIST set?
- asperous 9y agoI was thinking they might have "overfit" by picking hyper-parameters that happened by chance to result in a high CV score but wouldn't perform well in a holdout set. I could be wrong though, that's just my intuition.
- onetwotree 9y ago17 subjects per group is extremely small. Looking at it either from a machine learning or statistical point of view, using such a small sample is problematic. This is the chronic issue with fMRI studies, since administering an fMRI is extremely expensive, and has led to some very difficult to reproduce results in the field.
- killjoywashere 9y agoI believe some pretty fundamental fMRI spatial autocorrelation functions have been called into question as well (1). Sounds a bit like PowerPoint's autocontent wizard. (1) http://www.pnas.org/content/113/28/7900.abstract http://www.pnas.org/content/113/28/7900.abstract
- matt4077 9y agoOh, but the paradigm-shifting revolution of a paper, sorry: "poster" surely is "Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon": http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf
- matt4077 9y agoPeople love the "n=XX is far too little data!" argument, yet it's more complicated than that. Sometimes 600,000 is too little, yet sometimes 17 is enough. Example: you believe a newly found plant species is toxic. You give it to 17 "grad students volunteers", while giving a placebo to 17 others. All in the first group die aa gruesome death within 20 hours. None of the others do. Result: yes significance. (also: tenure!) I'm not saying that this study is significant (the statistics seem to be slightly beyond my event horizon), and your criticism also stops short of an outright dismissal of the research. But sample size alone makes for a bad measure of quality. Yes, even p-values are better.
- jacalata 9y agoI really hope you wouldn't get tenure for a study that killed all your subjects.
- bobstaples 9y agoin the paper they state they performed " 34 “leave one participant out” cross-validation cycles (folds)", the dataset still seems to small to be able to draw any conclusion though.