4 ms·

Look, I'm not a domain expert in the medical side of this, but the paper says they used a dataset, described in it's referenced paper as: "This challenge lever

by feral 7y ago

Look, I'm not a domain expert in the medical side of this, but the paper says they used a dataset, described in it's referenced paper as: "This challenge leveraged a database of dermoscopic skin images from the ISIC Data Archive1 , which at the time of this publication contains over 10,000 images collected from leading clinical centers internationally, acquired from a variety of devices used at each center. The images are screened for both privacy and quality assurance. The associated clinical metadata has been vetted by recognized melanoma experts. Broad and international participation in image contribution ensures that the dataset contains a representative clinically relevant sam- ple". [Gutman et al.] This paper selects a subset relevant to a certain condition (beyond my expertise). But that sounds pretty good to me. If the test set in the paper is carefully randomly selected, as the authors described (and I've no reason to disbelieve them) then performance on test set should be a good proxy to actual performance on unseen data, as the underlying dataset is designed to be representative of clinical practice. Of course that's not the same as having a clinically useful product. But commenters here are knocking these cancer researchers as if they are idiots, and saying very harsh things about their methods, when it seems like 3 minutes reading their Dataset section gives reasons to think their setup is actually pretty ok?

argonaut 7y ago
> 10,000 images collected from leading clinical centers internationally Lets say the data was collected from 5 different clinical centers. One risk is basically that when you deploy the model, it only works at those clinical centers due to idiosyncrasies specific to those centers. Or suppose certain doctors were more likely to take melanoma images, and certain doctors were more likely to take non-melanoma images, and both sets of doctors used different techniques to take images. These are just some ideas, there could be any number of confounding factors. Basically, one should be default suspicious of most research papers working with small datasets (the test set is only 100 images - this is very small) that have not been deployed in the real world or otherwise validated independently. The root comment of this thread is basically saying this (a different comment is the one calling the researchers idiots).
- feral 7y ago
  > These are just some ideas, there could be any number of confounding factors. There _could_ be. But when the source dataset was carefully gathered for a competition and the academics are saying "Broad and international participation in image contribution ensures that the dataset contains a representative clinically relevant sample", talking about multiple different equipment and labs, things look pretty promising. A lot of ML systems are built with much less rigourous datasets and do a good job when you put them in production. 10k such images were gathered from this process. The authors then randomly select 100 images from it to use as a test dataset. 100 is a small number. But that smallness is not relevant to the _selection_ issues here. Its only relevant to whether the measurements of performance are statistically significant (i.e. that we didn't end up with a sample that by chance is particularly favorable to the ML approach.) 100 data points is enough that that's unlikely (though one should check.) Additionally the authors talk about using a much larger validation set, and performing multiple runs and checking the validation accuracy is similar. Unless they deliberately left out the damning fact that their accuracy was a lot _higher_ on their small test set than on their validation set, then its even more unlikely that the test set is a sample that is particularly favorable to the ML approach. You could argue it happens to be particularly _unfavorable_ to the humans, but that seems a stretch. Perhaps they should have created multiple test sets, and a couple of different batches of human raters etc. But honestly, their setup seems pretty good to me. > (a different comment is the one calling the researchers idiots). That's fair.
  - argonaut 7y ago
    Their validation set consisted of 210 positive images. The test set consisted of 20 positive images. These are very small evaluation sets for deep learning. My point is the work is promising but should be viewed with healthy skepticism (by default). I would really not read anything in particular into "Broad and international participation... ...sample." That's just a claim in a paper, it's not "the truth".
    - feral 7y ago
      > These are very small evaluation sets for deep learning. Evaluation is a statistics question, and it doesn't matter that the deep learning model used is high capacity and needs a lot of training data. There's nothing inherently wrong with validating a complex model on a small amount of data. The paper has a section 4.2 that gives a statistical analysis. Granted, it'd be nicer if they had enough data to show statistically significant differences.

by feral 7y ago

argonaut 7y ago

> 10,000 images collected from leading clinical centers internationally Lets say the data was collected from 5 different clinical centers. One risk is basically that when you deploy the model, it only works at those clinical centers due to idiosyncrasies specific to those centers. Or suppose certain doctors were more likely to take melanoma images, and certain doctors were more likely to take non-melanoma images, and both sets of doctors used different techniques to take images. These are just some ideas, there could be any number of confounding factors. Basically, one should be default suspicious of most research papers working with small datasets (the test set is only 100 images - this is very small) that have not been deployed in the real world or otherwise validated independently. The root comment of this thread is basically saying this (a different comment is the one calling the researchers idiots).

feral 7y ago
> These are just some ideas, there could be any number of confounding factors. There _could_ be. But when the source dataset was carefully gathered for a competition and the academics are saying "Broad and international participation in image contribution ensures that the dataset contains a representative clinically relevant sample", talking about multiple different equipment and labs, things look pretty promising. A lot of ML systems are built with much less rigourous datasets and do a good job when you put them in production. 10k such images were gathered from this process. The authors then randomly select 100 images from it to use as a test dataset. 100 is a small number. But that smallness is not relevant to the _selection_ issues here. Its only relevant to whether the measurements of performance are statistically significant (i.e. that we didn't end up with a sample that by chance is particularly favorable to the ML approach.) 100 data points is enough that that's unlikely (though one should check.) Additionally the authors talk about using a much larger validation set, and performing multiple runs and checking the validation accuracy is similar. Unless they deliberately left out the damning fact that their accuracy was a lot _higher_ on their small test set than on their validation set, then its even more unlikely that the test set is a sample that is particularly favorable to the ML approach. You could argue it happens to be particularly _unfavorable_ to the humans, but that seems a stretch. Perhaps they should have created multiple test sets, and a couple of different batches of human raters etc. But honestly, their setup seems pretty good to me. > (a different comment is the one calling the researchers idiots). That's fair.
- argonaut 7y ago
  Their validation set consisted of 210 positive images. The test set consisted of 20 positive images. These are very small evaluation sets for deep learning. My point is the work is promising but should be viewed with healthy skepticism (by default). I would really not read anything in particular into "Broad and international participation... ...sample." That's just a claim in a paper, it's not "the truth".
  - feral 7y ago
    > These are very small evaluation sets for deep learning. Evaluation is a statistics question, and it doesn't matter that the deep learning model used is high capacity and needs a lot of training data. There's nothing inherently wrong with validating a complex model on a small amount of data. The paper has a section 4.2 that gives a statistical analysis. Granted, it'd be nicer if they had enough data to show statistically significant differences.