10 ms·
Deep learning outperformed dermatologists in melanoma image classification task
- assblaster 8y agoI've always thought that diagnostic-oriented specialties would be most at risk (pathology, dermatology, radiology, ophthalmology). As long as you have procedures, you will have a need for an extremely competent clinician that can synthesize all information and coordinate with use of hands or devices.
- rscho 8y agoAll the specialties you cite entail various manual procedures. Other than that yes, robots are not capable of replacing manual work in médicine yet.
- assblaster 8y agoExactly. As long as those specialities hold onto procedures, they'll be relatively ok.
- yumraj 8y agoI wonder if the results would be similar were the dermatologists to see the actual patient, in person, and then diagnose. And, then a photo was taken to be diagnosed by the CNN. In other words, while dermatologists may have been outperformed by deep learning in image classification, it is not evident if deep learning could do the same against dermatologist diagnosing in person. Also, not clear what the overall ratio of false negative/positives was in each case. Also, unless I missed it in the paper, I'd be curious to learn if the cases, in Fig. 4, where majority of humans and CNN disagreed, were the same where majority of humans disagreed too or not.
- nknealk 8y agoThis is extremely relevant. Factors like whether the patient has freckles or red hair often drive doctors to resect anything out of an abundance of caution. It doesn't look like these additional demographic factors were made available to clinicians. That said, the results here are still impressive. Source: I worked in clinical melanoma research for 2 years.
- ModernMech 8y agoMaybe? It's hard to say. I had a question about a mole once and what they did was take a photo of it with a special camera apparatus and they sent the image off to be diagnosed while I waited around. The doctor looked at it personally but it seems like the actual diagnosis was made by someone who never even saw me.
- sgt101 8y agoSame for me, the doctor I saw did say (after 3 seconds) "it's not a cancer but we'll take a picture anyway to be sure." I wonder if the same sort of thing would have happened if there was any real question in his mind?
- canada_dry 8y agoThis is a great example of where we need to get humans out of the equation when (if) a machine is conclusively proven to perform consistently better. It was justified (cost wise) to replace many human labourers on auto assembly since machines don't get tired, need breaks, have off days. It could certainly be argued it is even more important in the field of health care (reduce costs and improve outcomes) for all forms of image scanning.
- assblaster 8y agoMachines will never replace dermatologists, machines will only make them more efficient.
- djg123 8y agoWell... Never say never. If you said machines would not replace dermatologists in the next 1 or 2 decades, I would agree with you. After that, all bets are off.
- hhs 8y agoInteresting point. The authors also seem to note this in their conclusion: “Our findings suggest that artificial intelligence algorithms may successfully assist dermatologists with melanoma detection in clinical practice which needs to be carefully evaluated in prospective trials.” They use the verb assist. I think this will be the case, tools like this will be made available to board certified dermatologists to enhance their work. It won’t replace them anytime soon. This field continues to be among the most competitive for physicians in the US. The barriers to entry are not only high, the number of residency spots get smaller each year when compared to total applicants. If interested, here’s some stats on med school rankings and derm match results: https://escholarship.org/uc/item/59p3z80r https://escholarship.org/uc/item/59p3z80r. But over many years I wonder what will happen. Will there be paradigm shifts that make us rethink about all these specialities and subspecialties? Should we combine them or do something else?
- feral 8y agoIf you make a dermatologist 5x more efficient, don't you replace 80% of them? Or even better, allow them to spend more time on the hardest cases. And allow people with no access to a dermatologist now, access to a machine almost as good?
- JoshTko 8y agoIt's hard to imagine an narrow image classification task that humans will be able to beat NNs.
- hprotagonist 8y agoIt's not clear that melanoma diagnosis is fairly described as a static image classification task. Certainly it's a component, but (for example) patient history is, too.
- sgt101 8y agoConsidering that the ground truth comes from humans I would say that humans always outperform NN's and that results which show otherwise are demonstrating the limitations of the data set or testing process.
- leesec 8y agoIt's been proven that using humans as ground truth you can ultimately build a NN off that data which outperforms the humans.
- sgt101 8y agoWhat is meant by proven? What is meant by "the humans"? And how can any human say that is so?
- woadwarrior01 8y agoBecause an ensemble of humans can usually outperform a single human (quite like in artificial neural nets). One way to do it would be to setup a sufficiently large majority-voting ensemble of human classifiers.
- leesec 8y agoI mean, for example, you take 5 expert radiologists, and average their assessments when scoring an image, and your train a NN to predict those averaged scores, and with enough data the NN will beat any single selected doctor in accuracy.
- hprotagonist 8y agoAs always, let's see how well it does in live images. This system outperformed dermatologists on its own validation set of 100 images, which I would encourage you to interpret as "heartening preliminary evidence" but not much more. Posting high scores on your validation set is only as informative as your val set is representative of the real world. 70% specificity, 84% sensitivity looks OK on paper (maybe -- as another poster noted, it's equally fair to say it's good evidence that image-only diagnosis is bad no matter what does it), but it doesn't always feel that way in practice. As a cheap example, your word error rate for a speech recognition system has to be extremely low in order for that system to be nice to use -- way lower than most otherwise acceptable looking scores. This analogy only gets you so far, and i don't mean to impugn this study's test set, but another example is just because you can post 99.9% on MNIST doesn't mean that your system will approach that level of accuracy on digit recognition in the wild.
- learntoplay 8y agoIsn’t DeepMind about to release a medical product that will do something very similar to this? Right now I wouldn’t doubt how well these systems can perform as compared to trained specialists that rely on their eyes even for reading test results.
- rscho 8y agoThe question is whether we really measure things that are relevant and specific to the diagnosis. As of today the answer is largely no, which is why you cannot replace doctors yet and the reason why "90% of diagnoses are made on patient history".
- sgt101 8y agoI wonder if these products will have to go through proper trials like drugs do? If not, why not?
- hprotagonist 8y agohttps://www.fda.gov/medical-devices/digital-health/software-medical-device-samd https://www.fda.gov/medical-devices/digital-health/software-... https://www.fda.gov/medical-devices/ivd-regulatory-assistance/overview-ivd-regulation https://www.fda.gov/medical-devices/ivd-regulatory-assistanc...
- gigantum 8y agoInteresting, however there is no indication from the publisher or researchers how this result can be reproduced. It's nice that they put in some of the training data, but imagine how much more impactful to the community this could be if those interested could reproduce - and iterate - on this... At Gigantum (https://github.com/gigantum/gigantum-client https://github.com/gigantum/gigantum-client) this is literally our raison d'être to make this process as simple as possible.
- mikehollinger 8y agoI find it most interesting to use tools like this to augment - not replace diagnosticians and specialists. Of course the key here is that the training set is crucial to building a high quality model - which of course needs a set of specialists to give their consensus on the diagnosis of the patient based on the images. Presuming those folks can agree - the technology becomes a force multiplier for good. If they disagree or label things problematically - they become a force multiplier for bad.
- deleted 8y ago[deleted]
- sgt101 8y agoSo long as the system is implemented in such as way as to stop it becoming a crutch or a default. If it's a tool in the flow then I think such things (not this one given the expert review information in this thread?) might be very valuable.
- georgeek 8y agoEvery doctor tends to have a very static sensitivity-specificity preferences (true positive rate aka recall and true negative rate, respectively). One of the interesting consequences of using an automated diagnostic tool (already mentioned in Esteva et al 2017's Nature article) is that the sensitivity level can be chosen dynamically, depending on additional risk factors.
- leelin 8y agoMaybe a dumb question from a non-medical guy: are medical images considered "stationary" from a stats viewpoint? That is, will medical images of diseases we diagnose in the next 20 years look a lot like the ones from the past 20 years, or is there a danger of over-fitting on an evolving data set? Could either the technology or the biology of the disease evolve? In a prior life I was a quant trader, and financial market data is notorious for having the non-stationary problem. On top of market rules and structures changing all the time, once someone discovers a profitable trading idea, their own actions change what the data looks like for everyone else from that point forward.
- JoshTko 8y agoMy guess is largely no. The diagnosis and actions taken are in isolated environments e.g, a cancer in person B is not impacted by a diagnosis and treatment in patient A. This is not the case in trading where everyone is analyzing and influencing a single messy environment.
- chasedehan 8y agoThis isn’t time series data so don’t need to worry about stationarity.
- notahacker 8y agoOverfitting to people with certain complexions of healthy skin seems like a potentially much bigger problem than evolution of ailments.
- 1wheel 8y agoYup, this thread has a nice overview of ways performance on a validation set can overestimate clinical performance: https://twitter.com/IAmSamFin/status/1122271463170564100 https://twitter.com/IAmSamFin/status/1122271463170564100 Another example of change over time: > One difficulty in such a comparison is that Gleason grading standards have shifted over time, so that scores below six are now rarely assigned, and assigning a higher grade has become more common https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3775342/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3775342/
- 8y ago
- weaklearner 8y agowell the answer is a bit more complicated than just replacing dermatologists with a CNN. I am pretty convinced the better approach is something like the approach in this paper-use the CNN on easy cases-have the CNN tell a human which instances are hard to classify. Many images are easy to classify-but some are hard [even for the cnn] and humans should give those images more study. https://arxiv.org/abs/1903.12220 https://arxiv.org/abs/1903.12220
- michaelhoffman 8y agoReporting only sensitivity/specificity/ROC metrics and not reporting precision/positive predictive value is a bad sign. Especially since the latter is what health systems will want to look at before deciding on implementation. The fact that they fiddled with the balance of classes in the test set makes the above even worse.
- StreamBright 8y agoDeep learning outperformed humans in random pattern recognition task using a dataset that somewhat similar to live data.
- jmpman 8y agoAny place to upload an image?
- bobowzki 8y agoIf this is regarding yourself just make an appointment with a dermatologist. Source: I'm and MD.
- 0xab 8y agoI do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the wrong things that reward the network. Because the dataset is imbalanced you can't use an ROC curve, sensitivity, or specificity. You need to use precision and recall and make a PR curve. This is machine learning and stats 101. * They measure the wrong thing about humans. What a doctor does is they decide how confident they are and then they refer you to a biopsy. They don't eyeball it and go "looks fine" or "it's bad". They should measure how often this leads to a referral, and they'll see totally different results. There's a long history in papers like this of defining a bad task and then saying that humans can't do it. * They have a biased sample of doctors that is highly skewed toward people with no experience. Look at figure 1. A lot of those doctors have about as much experience to detect melanoma as you do. They just don't do this task. * "Electronic questionnaire"s are a junk way of gathering data for this task. Doctors are busy. What tells the authors that they're going to be as careful for this task as with a real patient? Real patients also have histories, etc. I could go on. The number of problems with this paper is just interminable (54% of their images were non-cancer because a bunch of people looked at them. If people are so wrong, why are they trusting these images? I would only trust biopsies). This isn't coming to a doctor's office anywhere near you. It's just a publicity stunt by clueless people. Please collaborate with some ML folks before publishing work like this! There are so many of us!
- arturadib 8y agoAgreed. See this paper for a reputable reference in this space: https://www.nature.com/articles/nature21056 https://www.nature.com/articles/nature21056
- claytonjy 8y agoWhy would you ever balance your test data? If 80/20 is the actual population distribution, the sample that forms your test set should conform to that. Balance all you want in train/validation sets, but never the test set. Not balancing and using ROC is a terrible combo, but the metric is the problem, not the lack of artificial balance.
- deleted 8y ago[deleted]
- sonnyblarney 8y agoDespite the issues listed below, I believe this will be the future of such classifications. In the future your doctor will have an image scanner in their office and you'll get 'really cheap' diagnosis of this to back up the doctors opinion. Then you'll go for biopsy etc..
- mrosett 8y agoGiven all of the other concerns raised by commenters in this thread, I wouldn't be surprised to find that there's some sort of data leakage as well.
- aaron695 8y agoCall me crazy but the fact people can't record spots in their body and how they change over time might be step one. For humans and AI. Why don't I have a record of my spots from being a baby? Privacy left to implementing, but there's a few ways you could go.
- bonyt 8y agoI'm always wary of claims like these, since it is difficult to get a clean dataset of medical information, and it is difficult or impossible to know for sure what exactly what the classifier is looking at to classify the image: > For example, Roberto Novoa, a clinical dermatologist at Stanford University in the US, has described a time when he and his colleagues designed an algorithm to recognize skin cancer – only to discover that they’d accidentally designed a ruler detector instead, because the largest tumours had been photographed with rulers next to them for scale. Source: https://physicsworld.com/a/neural-networks-explained/ https://physicsworld.com/a/neural-networks-explained/
- gbronner 8y agoMelaFind did this 15 years ago, and had a database of 50k lesions. Its classifier did about as well as derms, and much better than GPs. It got FDA approval, but, by that time, they had run out of money.