10 ms·
Computers do better than pathologists in predicting lung cancer type, severity
- clumsysmurf 10y agoMy friend worked in a pathology lab, run by a recognized world expert in the field (wrote college texts, gave lectures, etc) yet the performance was 'mediocre' due to reoccurring mistakes made by humans: * Doctors disagreed with each other frequently. Most senior won. * Results from one patient were given to another patient * ... or results were lost entirely * Lab destroyed samples before examination or stained incorrectly * Samples never received, lost in mail or never picked up from airport These folks did have a tough job, looking into microscopes for many hours per day. At some point I imagine things just start looking the same as fatigue sets in.
- huherto 10y agoMay be the next medical breakthrough is really going to be TODO lists. Which reminds me of Lotus Notes. 20 years ago it widely used to automate workflows. But I haven't seen anything similar widely used since.
- maxerickson 10y agoYou've heard of The Checklist Manifesto? Medicine features in it.
- ddeck 10y agoIt already is: From: The effects of safety checklists in medicine: a systematic review: [1] Safety checklists appear to be effective tools for improving patient safety in various clinical settings by strengthening compliance with guidelines, improving human factors, reducing the incidence of adverse events, and decreasing mortality and morbidity. None of the included studies reported negative effects on safety. [1] http://www.ncbi.nlm.nih.gov/pubmed/24116973 http://www.ncbi.nlm.nih.gov/pubmed/24116973
- derefr 10y agoTo me, this is a generalization of the benefit of software/automation: what we're really discovering is that when we force people to think through all the edge-cases in a process in order to explain that process to something as dumb as a computer—or a checklist—we dig up a lot of implicit institutional knowledge that, crucially, not everyone has. (Everybody that had it just assumed everyone else knew!) The result of such a process can be a checklist, or a workflow diagram, or a program. The form of it isn't really important, so much as the fact that the business logic being applied (by a human or a computer) to each case now embeds all the previously-scattered knowledge of the domain explicitly, making each node applying it behave as the union of its institution's knowledge-bases, rather than as the intersection of those knowledge-bases.
- tormeh 10y agoAlso, a checklist doesn't get tired and forget a step.
- Zigurd 10y agoProtocol books formalize checklists for ER medicine, over-the-phone triage, etc.
- delazeur 10y ago> Doctors disagreed with each other frequently. Most senior won. I'm told this is a huge problem in medicine. They might be even more hierarchical than the military.
- Fede_V 10y agoI am surprised they only used ~2000 samples. That's a very tiny amount of data to train a decently sized CNN, even with data augmentation. I presume using a pretrained net probably won't help you since the features found in pathology slides are so unlike those of natural images.
- yread 10y agoEach image can be multiple gigabytes so it could be computationally difficult. Also those are probably all the lung cancer slides that TCGA has (edit: there seems to be 3500+ lung cancer slides at [1] so I was wrong. Maybe they're not all H&E stained) [1] http://cancer.digitalslidearchive.net/ http://cancer.digitalslidearchive.net/
- acveilleux 10y agoYou also need to retain a significant set of slides the classifier wasn't trained with so you can verify that it does work for data outside the training set.
- dave_sullivan 10y agoI've done a bit with deep learning and medical images (brain, lung, skin). One thing that surprised me, but kind of makes sense, is the net seemed to perform well without a ton of training data. I think it might actually be easier for the model because the images are much more similar than in a "natural image" setting where there are more degrees of freedom in the subject itself.
- shoyer 10y agoSomewhat surprisingly, the paper doesn't actually use deep learning.
- dandermotj 10y agoEveryone is missing this. They used simple random forests and survival models with basic k-fold cross validation. There's no breakthrough in ML here!
- Strilanc 10y agoYet another press release that fails to reference the study it's talking about. Not to mention giving no quantitative details about its lead ('Trounced'? By how much?). The study is "Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features" by Kun-Hsing Yu et al. It's open access: http://www.nature.com/ncomms/2016/160816/ncomms12474/full/ncomms12474.html http://www.nature.com/ncomms/2016/160816/ncomms12474/full/nc...
- daveguy 10y agoComputers trounce* pathologists... * by an un-quantified amount using only histology slides -- a limitation not enforced in actual medical practice.
- cicero19 10y agoyeah.. seriously.. what academic journal would accept that kind of statement.
- ChristianBundy 10y agoNature, apparently.
- frozenport 10y agoIndeed, this is a press release and not journalism.
- muxxa 10y agoI wonder how the pathologists would fare if they were also put through the same process as the ANN, i.e. given training data along with immediate feedback on whether their prognosis was right or wrong, then tested on the reserve data. Pathologists give daily prognosises but only get feedback, if at all, many years later.
- 50CNT 10y agoI don't know much about the medical field, so excuse my ignorance in this, but couldn't we train Pathologists on the same data we train ANNs on? That way we could establish similar speed in feedback loops.
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- mmierz 10y agoI think this is already done. Pathologists-in-training have to learn somewhere. Machine learning researchers didn't exactly invent the concept of studying practice problems with provided answers, then taking a test.
- Gatsky 10y agoThis is actually the crucial point for me. The care and effort put into training the ANN likely far exceeds the care and effort put into training the pathologist. Pathologists learn their craft like most doctors do - a mixture of consulting text books and supervision by seniors, but mostly just by working it out themselves. Pathologists also tend to optimise for dealing with rare and unusual cases rather than small incremental gains in their performance on routune diagnoses, because of how they are assessed in exams (ie more breadth than depth of knowledge). This gives them good 'general AI' for whatever case walks in the door, rather than a highly constrained test set of one type of case.
- charlesism 10y agoThe world is full of constant hype about "breakthrough" studies, and cures for diseases that are "coming soon". Somehow the cures seldom materialize, and too often, if anyone bothers to check, they can't replicate the studies. I wonder if the secret sauce that makes science effective isn't so much the scientific method, as the spirit of skepticism behind it. Without that, it's easy to make errors. And wishful thinking is pervasive because there are strong incentives: career advancement, corporate influence, ego, and so on. My view, assuming AI doesn't kill us, is that it's going to save all of our asses (from ourselves).
- bbctol 10y agoWhile I have serious reservations, I do appreciate philosopher Paul Feyerabend's idea of science as "epistemological anarchism." In the end, all rule-based concepts of how to accumulate knowledge and understanding fall victim to methodological problems: the only way to really do science is a no-holds-barred, anything goes attitude of a mind against the universe.
- aschampion 10y agoI worked on a similar pipeline for renal cell carcinoma a few years ago [1], although we only published a small subset of results since parts of the pipeline (e.g., finding representative tiles, survival prediction) had better results being produced elsewhere in the lab. Regarding the hook in the headline -- computers surpassing pathologists -- it's a bit like automated driving in that even if true the immediate problem is the social and economic system. That is, we're not going to be removing the pathologist from the diagnostic and prognostic process anytime soon for many reasons, so how instead do we leverage machine learning in concert with the human observer to improve the diagnostic system? For that reason, decision introspection may be as valuable a topic of research as improving classification accuracy: justifying a particular automated classification to the pathologist, directing them to representative regions, and describing regions of feature space in biological terms. [1] http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6945104 http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6945104
- mgreg 10y agoVery good point. Even if this particular study isn't a "breakthrough" it is likely a matter of time before AI / Deep Learning (what term am I supposed to use?) is able to better interpret visual data than a human. The real barriers to adoption of such a technology are significant and can be witnessed in the popular Self Driving Car technology race. Many people seem to believe the technology problem has already been solved or will be shortly but regulations, public perception, and adjacent industries (i.e. insurance) must adapt before we actually see self driving cars for the common man. These tend to be slow moving beasts. Given how many applications we're seeing for AI it seems that there may be a battle for attention of lawmakers and others for which AI application we put into production first given the limited bandwidth our regulators have; it's not like congress has been especially productive over the last many years.
- _0ffh 10y agoThis! As far as I can remember, the experience in chess is that a human+computer team beats either of them working alone (sorry, can't find a reference just now).
- WhitneyLand 10y agoWhy isn't this already is practice given ML advances in recent years? Why are mammograms still being subjectively interpreted? Are there not already companies today where you can upload medical imaging results and get back results that beat humans?
- chestervonwinch 10y ago> Why are mammograms still being subjectively interpreted? Computer-aided detection and diagnosis software is used in clinical practice, e.g. see [1]. I believe that false-postives-per-scan is still somewhat of an issue. [1]: http://www.hologic.com/products/imaging/mammography/image-analytics http://www.hologic.com/products/imaging/mammography/image-an...
- mgreg 10y ago>Why isn't this already is practice given ML advances in recent years? Well it would be nice to first validate the results somewhere else (peer review should actually be completed) and no doubt some real world trials proceed and their results evaluated. This is what any FDA approval, at a minimum, would entail. Keep in mind what the author's themselves note in how this study may differ from the "real world": "One limitation of this study is that cases submitted for TCGA and TMA databases might be biased in terms of having mostly images in which the morphological patterns of disease are definitive, which could be different from what pathologists encounter at their day-to-day practice." Putting this into practice also requires looking at the bigger picture; not just the accuracy of the diagnosis vs. humans. For example I have found that in studies doctors are much more conservative in their diagnosis than an academic algorithm with no implication on patient outcome. For instance if the question is a) "cancer" or b) "not cancer" the doctor, fearing patient harm, malpractice suits, career derailment, will be biased to identifying "a) cancer" because the perceived costs of a false positive (treating for a nonexistent cancer which may have significant ill effects) are lower than a false negative (not treating a real cancer leading potentially to death). This will reduce the human doctor's accuracy on diagnosis. What the "right" bias is in an individual case can bring in many more factors and moral questions out of scope for this study. This is not to argue that tools like these should not be developed and utilized but practical application is often more difficult than just solving the technology challenge. I'm certain that we'll see many advances in healthcare due to ML.
- njharman 10y agoUm, computers do better than humans in the vast majority of applications to which they are programmed for. By which I mean, the news is not that computer > human, but that medical industry/profession/jobs are taking first baby steps at being automated away.
- noobermin 10y agoHow soon until computers can make better MVC apps than people?
- WhitneyLand 10y agoNo, I think any task where computers are newly enabled to beat humans is newsworthy. I'd like to know about each of the dominoes as they continue to fall.
- danielrm26 10y agoHuman error comes from a lot more than just pure assessment of whether something is cancer or not. Fatigue, willpower drain, emotion, and many other factors can make a top-tier human expert into a mediocre one (or worse) in a way that is very hard to detect. Even if computers aren't perfectly matching the ideal human (which they will inevitably pass, of course) they're still massively valuable in that they can maintain their level of expertise when humans cannot.
- deleted 10y ago[deleted]
- yread 10y agoIt's nice that the article is open access and even includes the R code use to perform the analysis [1] http://www.nature.com/ncomms/2016/160816/ncomms12474/full/ncomms12474.html http://www.nature.com/ncomms/2016/160816/ncomms12474/full/nc... [2] http://www.nature.com/article-assets/npg/ncomms/2016/160816/ncomms12474/extref/ncomms12474-s5.txt http://www.nature.com/article-assets/npg/ncomms/2016/160816/...
- shazam 10y agoDoes anyone know about how to get involved in work like this? I'm outcome-oriented so disease diagnosis would be motivating for me. I'm looking to help out on an open-source project along these lines.
- teekert 10y agoMaybe help improve http://ilastik.org/ http://ilastik.org/ ?
- yread 10y agohttp://cellprofiler.org/ http://cellprofiler.org/ is open source and was used for this paper
- histowiz 10y agojoin Histowiz, we have 10x the data! It's one of the largest preclinical pathology slide database plus metadata like biomarkers, organs, disease, outcome, experimental conditions...to do correlations with histopathology image features
- nikcheerla 10y agoThere's a LOT of biological data out there. With a little bit of effort, researchers can design AI systems that beat doctors at most specialized tasks, in both computer vision and genetic data analysis. The question is how such systems can actually be used in practice.
- tvladeck 10y agoThe article mentioned they only had a dataset of 2,186 slides. Isn't this a _tiny_ dataset to do any learning on images? They also mentioned they blew up the predictors to over 10,000 features. This sounds like a recipe for overfitting to me. I can't access the actual paper (getting a 500 error), but does anyone know how they built these algorithms?
- niels_olson 10y agoThe images are 1-5 gb each. They tile them. Then they did some simple sorting (remove all tiles with lots of white), (emphasize tiles with lots of blue), then they did random forests, survival models and k-fold cross validation. The test set is interesting: they used a completely unrelated dataset of tissue microarray spots. This are tiny, 1-2 mm circles. Whereas the TCGA images are full size tissue cassettes, up to 1" on the short side.
- naveen99 10y agoPathology slides are analogue data (lot of data, more than radiology scans). Even tertiary care hospitals do not routinely digitize slides unless they have to be returned to another hospital or used for a teaching conference.