3 ms·

As always, let's see how well it does in live images. This system outperformed dermatologists on its own validation set of 100 images, which I would encourage

As always, let's see how well it does in live images. This system outperformed dermatologists on its own validation set of 100 images, which I would encourage you to interpret as "heartening preliminary evidence" but not much more. Posting high scores on your validation set is only as informative as your val set is representative of the real world. 70% specificity, 84% sensitivity looks OK on paper (maybe -- as another poster noted, it's equally fair to say it's good evidence that image-only diagnosis is bad no matter what does it), but it doesn't always feel that way in practice. As a cheap example, your word error rate for a speech recognition system has to be extremely low in order for that system to be nice to use -- way lower than most otherwise acceptable looking scores. This analogy only gets you so far, and i don't mean to impugn this study's test set, but another example is just because you can post 99.9% on MNIST doesn't mean that your system will approach that level of accuracy on digit recognition in the wild.

learntoplay 8y ago
Isn’t DeepMind about to release a medical product that will do something very similar to this? Right now I wouldn’t doubt how well these systems can perform as compared to trained specialists that rely on their eyes even for reading test results.
- rscho 8y ago
  The question is whether we really measure things that are relevant and specific to the diagnosis. As of today the answer is largely no, which is why you cannot replace doctors yet and the reason why "90% of diagnoses are made on patient history".
- sgt101 8y ago
  I wonder if these products will have to go through proper trials like drugs do? If not, why not?
  - hprotagonist 8y ago
    https://www.fda.gov/medical-devices/digital-health/software-medical-device-samd https://www.fda.gov/medical-devices/digital-health/software-... https://www.fda.gov/medical-devices/ivd-regulatory-assistance/overview-ivd-regulation https://www.fda.gov/medical-devices/ivd-regulatory-assistanc...
feral 8y ago
First off, if I'm reading correctly, it outperformed on its test set. This is different as it doesn't get to see that at any point before it's final. If the authors have done a diligent job here, that should be good evidence of it's accuracy. It's also encouraging to see they do multiple training runs, getting similar accuracy, and that their ROC is generally better than not just the average physician, but almost all.
- sgt101 8y ago
  Remember the test set is derived from the same source as the training set. This is not the case in the wild.
  - Florin_Andrei 8y ago
    Yeah, this is the ML equivalent of "it works in vitro".
    - hprotagonist 8y ago
      That's a lovely way of putting it. You're exactly right.
      - feral 8y ago
        Look, I'm not a domain expert in the medical side of this, but the paper says they used a dataset, described in it's referenced paper as: "This challenge leveraged a database of dermoscopic skin images from the ISIC Data Archive1 , which at the time of this publication contains over 10,000 images collected from leading clinical centers internationally, acquired from a variety of devices used at each center. The images are screened for both privacy and quality assurance. The associated clinical metadata has been vetted by recognized melanoma experts. Broad and international participation in image contribution ensures that the dataset contains a representative clinically relevant sam- ple". [Gutman et al.] This paper selects a subset relevant to a certain condition (beyond my expertise). But that sounds pretty good to me. If the test set in the paper is carefully randomly selected, as the authors described (and I've no reason to disbelieve them) then performance on test set should be a good proxy to actual performance on unseen data, as the underlying dataset is designed to be representative of clinical practice. Of course that's not the same as having a clinically useful product. But commenters here are knocking these cancer researchers as if they are idiots, and saying very harsh things about their methods, when it seems like 3 minutes reading their Dataset section gives reasons to think their setup is actually pretty ok?

learntoplay 8y ago

Isn’t DeepMind about to release a medical product that will do something very similar to this? Right now I wouldn’t doubt how well these systems can perform as compared to trained specialists that rely on their eyes even for reading test results.

rscho 8y ago
The question is whether we really measure things that are relevant and specific to the diagnosis. As of today the answer is largely no, which is why you cannot replace doctors yet and the reason why "90% of diagnoses are made on patient history".
sgt101 8y ago
I wonder if these products will have to go through proper trials like drugs do? If not, why not?
- hprotagonist 8y ago
  https://www.fda.gov/medical-devices/digital-health/software-medical-device-samd https://www.fda.gov/medical-devices/digital-health/software-... https://www.fda.gov/medical-devices/ivd-regulatory-assistance/overview-ivd-regulation https://www.fda.gov/medical-devices/ivd-regulatory-assistanc...

feral 8y ago

First off, if I'm reading correctly, it outperformed on its test set. This is different as it doesn't get to see that at any point before it's final. If the authors have done a diligent job here, that should be good evidence of it's accuracy. It's also encouraging to see they do multiple training runs, getting similar accuracy, and that their ROC is generally better than not just the average physician, but almost all.

sgt101 8y ago
Remember the test set is derived from the same source as the training set. This is not the case in the wild.
- Florin_Andrei 8y ago
  Yeah, this is the ML equivalent of "it works in vitro".
  - hprotagonist 8y ago
    That's a lovely way of putting it. You're exactly right.
    - feral 8y ago
      Look, I'm not a domain expert in the medical side of this, but the paper says they used a dataset, described in it's referenced paper as: "This challenge leveraged a database of dermoscopic skin images from the ISIC Data Archive1 , which at the time of this publication contains over 10,000 images collected from leading clinical centers internationally, acquired from a variety of devices used at each center. The images are screened for both privacy and quality assurance. The associated clinical metadata has been vetted by recognized melanoma experts. Broad and international participation in image contribution ensures that the dataset contains a representative clinically relevant sam- ple". [Gutman et al.] This paper selects a subset relevant to a certain condition (beyond my expertise). But that sounds pretty good to me. If the test set in the paper is carefully randomly selected, as the authors described (and I've no reason to disbelieve them) then performance on test set should be a good proxy to actual performance on unseen data, as the underlying dataset is designed to be representative of clinical practice. Of course that's not the same as having a clinically useful product. But commenters here are knocking these cancer researchers as if they are idiots, and saying very harsh things about their methods, when it seems like 3 minutes reading their Dataset section gives reasons to think their setup is actually pretty ok?