3 ms·
"model was trained on 84,743 retinal fundus photos from the UK Biobank dataset. External validation was performed on 252 fundus photos" Is it common practice i
by stewbrew 5y ago
"model was trained on 84,743 retinal fundus photos from the UK Biobank dataset. External validation was performed on 252 fundus photos"
Is it common practice in this field to test an overfitted model's performance with such a small data set so that the test could yield random results?
- hervature 5y agoThe probability of a 50% guess getting 75% accuracy (I believe the paper is something like 77.2%) on 252 trials is 1 in 10^15.
- stewbrew 5y agoGood point but you should probably use the prob that it's 75% or higher.
- drdeca 5y agoAre you asserting that it is overfit? Also, before they tested on the other smaller dataset from a different source, aiui, they also trained only on the earlier subset of the first source, and used the later portion from the first source (with no overlap in patients) for the testing. (also, I'm not sure that 252 is really all that small?)
- avalys 5y agoThat was not the only validation set they used.
- make3 5y agoWhy do you say that the model is overfitted? You have no way of knowing that. Plus, 84 743 is a very reasonable size for a vision dataset with a binary prediction