16 ms·
Use of artificial intelligence for image analysis in breast cancer screening
- fexelein 5y agoIt depends on the accuracy of the humans as well. Less accurate doesn’t mean not accurate.
- lend000 5y agoThe systems tested were from papers that claimed to be more accurate than radiologists. 1. Presumably we are most interested in the 6% that are more accurate. 2. The more interesting part, although not entirely surprising, is that 94% of research is overstated, at best.
- MattGaiser 5y agoA lot of the systems seem to be trained on very few images. I work for a ML commercialization company. People have brought us models trained on 10 tumors.
- malcolmgreaves 5y ago> 1. Presumably we are most interested in the 6% that are more accurate. Why? At 94% failure, this 6% sounds like lucky guessing by a computer.
- da_chicken 5y agoTitle isn't remotely clear here. "94% of 36 AI systems evaluated were less accurate" could mean the 36 systems were less accurate in 94% of cases, or it could mean that 34 of the 36 systems were less accurate. From the actual article's abstract, here is what was meant: > Thirty four (94%) of 36 AI systems evaluated in these studies were less accurate than a single radiologist, and all were less accurate than consensus of two or more radiologists. I wonder if the HN title was written by one of these AI systems.
- cratermoon 5y agoNo, I shortened the title because HN wouldn't allow the full one. I thought it would be pretty obvious that 94% of 36 is 34 (rounded).
- rightbyte 5y agoI don't know how much guessing is involved in making a diagnosis but some AIs could have been lucky. It would be hard for the doctor to win all 'games' even if better.
- macksd 5y agoI was a bit confused by exactly what the percentage in the headline was, so here: >> In two of the largest retrospective cohort studies of AI to replace radiologists in Europe (n=76 813 women), all AI systems were less accurate than consensus of two radiologists, and 34 of 36 AI systems were less accurate than a single reader I'm still a little unclear how you measure the accuracy of a consensus of radiologists, without relying on the consensus of other radiologists. Maybe just a bigger consensus, or by using known end results but looking at imagery that could have been an early warning?
- Areading314 5y agoThe methodology in this paper compared the results of the radiology report to a follow up biopsy.
- michaelmrose 5y agoYou could simply see if the person had actually gotten cancer with the benefit of perspective.
- honksillet 5y agoWell, to train the AI you would be using radiologist reads anyways so I'm not sure that's really a problems.
- cratermoon 5y agoOh yes, just that easy! Here, AI, he's some radiologist reads. Learn them, we're going live in Q1 2022.
- matchagaucho 5y agoOr inversely, can this be interpreted as 2 of 36 AI systems are better than humans? Therefore Hospitals should be investing in those 2 systems?
- dbt00 5y agoOnly if you started with a hypothesis that those two AIs were the best. If you did not, then you're just p hacking.
- nikita 5y agoWhat about the 6%. Are they better - let's use those!
- maweki 5y agoWe just need worse radiologists.
- jeffbee 5y agoUnder-rated comment here. The question is not whether some AI beats experts, even though the conclusion of this paper is clearly that the state-of-the-art AI does in fact beat them, but whether they are going to beat the radiologist at the free clinic in some poorly-served place.
- nradov 5y agoThe US healthcare system has little correlation between price and care quality. More expensive providers don't reliably delivery better patient outcomes.
- wolverine876 5y agoDo you know where there is more info on that?
- nradov 5y agoThis is a good start. https://journalofethics.ama-assn.org/article/complex-relationship-between-cost-and-quality-us-health-care/2014-02 https://journalofethics.ama-assn.org/article/complex-relatio... https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4863949/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4863949/
- 256lie 5y agoEven the average radiologist is high variable, not to mention inter-reader variability.
- avip 5y agoBut 100% of AI systems are more accurate than some radiologist.
- snovv_crash 5y agoNot necessarily. You overestimate what people try to pass off as AI.
- RosanaAnaDana 5y agoAn intern.
- timmattison 5y agoWell, just like all technology that wasn’t perfect from the start we should probably just give up. /s
- MattGaiser 5y agoI think the argument is more "we are not as far along as some might claim."
- cratermoon 5y agoOr, like perpetual motion machines or squaring the circle with compass and straightedge, perhaps the entire premise of the technology is flawed and unfixable.
- mcguire 5y agoOr maybe just stop claiming that the technology is already better. In other words, lying.
- karmasimida 5y agoBut they are probably like 90% cheaper, and 99% faster than a radiologist, not even going to talk about the throughput. It seems really good news to me actually.
- jacquesm 5y agoIf only it was as simple as that. In medicine a wrong diagnosis can have very bad real world consequences so 90% cheaper and 99% faster has a very real possibility of being a net negative.
- TameAntelope 5y agoThere are lots of ways of lightening the load with AI aid, though without fully removing the need for human eyes.
- jacquesm 5y agoYes, and these are typically the best systems, where the doctor is assisted rather than replaced. I've looked at several over the last years and the comparison doctor+augmentation tool to doctor alone and tool alone is invariably superior. For now I think that is the most productive and the safest route.
- 256lie 5y agoFor breast screening, this task is high volume and low prevalence and AI can help with radiologist burnout from increased caseload.
- make3 5y agoway cheaper than 90%, and way faster than 99%. Radiologists are crazy expensive, at least 50$ per scan. Let's say you use a 1000$ GPU (which you don't need to as these models are usually tiny in practice because of the limited size of medical datasets and because you're not doing hundreds of scans per second.. a CPU is very likely fine). Then.. just 20 scans in you make your money. Let's say the license cost is 5$ per scan. Still make your money very fast.
- yangl1996 5y agoThey could have said "35 out of 36" instead of putting some scary number like "94%".
- MattGaiser 5y agoIs that less scary?
- watwut 5y ago35 out of 36 is scary.
- qweqwweqwe-90i 5y agoYou mean 34/36
- cratermoon 5y agoI shortened the title to fit HN limits. Why do you think "34 out of 36 were wrong" is less scary than "94% were wrong"?
- rhacker 5y agoAnother way to think about that: %99.99 of AIs lose chess against our best human player. When in reality 1 AI consistently beats all Human players every time.
- cratermoon 5y agoOnly if you can go back and demonstrate that the 2 AIs that didn't fail are consistent and demonstrably correct. Otherwise they just got lucky.
- axiom92 5y ago35 out of 36 will be scarier to most people. In behavioral psychology they call this "denominator neglect." https://www.askattest.com/blog/guides/denominator-neglect https://www.askattest.com/blog/guides/denominator-neglect
- make3 5y agoTitle could be something like "Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy" (which is the original title)
- cratermoon 5y agoWould you have read an article with that title? Does that title accurately represent the main findings of the study?
- mcguire 5y agoAbsolutely, if you want to preserve AI research funding. It's not like we haven't done the AI Winter thing before. People should be used to this stuff by now.
- YossarianFrPrez 5y agoThis type of evaluation (both the meta-evaluation and the underlying studies themselves) are the exact sort things AI enthusiasts should want researchers to do. It cuts through the hype, and gives everyone a clear assessment of where the technology currently stands. Clearly there is room for improvement. Maybe this study will also spur the development of new types of systems which augment human/radiologist decision-making.
- snarf21 5y agoAI isn't a stand-alone panacea for this. We've done lost of studies with other radiologists. Taking the union of two "bad" radiologists outperforms a single "good" radiologist. Humans aren't machines, we have bad days. I think the long term benefit of this kind of AI is as an audit AFTER the radiologist has done their assessment. If there is a mismatch, it should trigger a follow-up with a different a radiologist (without telling them the result of the 1st or the AI) to build consensus.
- ajmurmann 5y agoThis! I wish we did this type of usage a lot more in a variety of areas. I previously proposed this at a job to enhance filtering out of false-positive security scan results. I think it doesn't get used because it would increase the immediate cost, not reduce it.
- tsjq 5y ago>I think it doesn't get used because it would increase the immediate cost, not reduce it. indeed that is the problem in many areas. Almost always mgmt looks with "tech = fewer human = instant cost savings" expectations. and anything that increases cost in the short term are shot down. sad . all blame on the cost-center vs profit-center mindset.
- spockz 5y agoI think getting a second “opinion” is pretty standard for biopsies?
- 5y ago
- Traster 5y agoI think this headline is bad. When you read the abstract you find that what this is really saying is that there are essentially no studies that are fit for purpose to evaluate AI systems for breast screening in general and those that are the closest to being suitable show the worst results. That's the real take-away here - we're not even at the point where we're collecting good data about AI accuracy, let alone actually producing accurate AI. I would suggest these two things might be linked - the only way you get publishable results is by failing to do rigorous studies.
- jszymborski 5y agoYou can get away with publishing models with worse accuracies if you can pitch a more fair evaluation scheme, but it's way harder, and you don't get to do flashy presentations where you state you're SotA. I hope I don't offend any other people in the field, but I think historically fields like histology/radiology/MRI screening via AI have been subject to less scrutiny by virtue of their multi-disciplinary nature... particularly in the past, it was hard to find reviewers that both (A) understood the biological/clinical validation (B) the technical validation. I think things have drastically changed over even the last 5 years, which is why you're more likely to see headlines like these. We have a lot of discussions like these during our lab's journal clubs. The optimists among us (e.g. myself) argue that this opens up a lot of opportunities whereby more fair validations/benchmarks allows us to compete with "SotA" methods that are actually fragile and easy to surpass on even ground. More pessimistic members are quick to note that it's far harder to introduce new, fairer validations and that reporting lower metrics is far less buzz worthy (all true points). This reminds me of a fundamental error that was recently found in methods that tried to predict protein-protein interactions. It was found that the train/test/validation method used by ostensibly every paper was leaking huge amounts of data [0]. When we plugged that leak, we saw much more modest metrics, but focusing on regularisation allowed us to beat the competition [1]. The kicker? The information leak were identified in 2012, yet you'll see papers written every year that have the same leak and report >90% accuracies, and >0.9 ROCs. [0] https://www.nature.com/articles/nmeth.2259 https://www.nature.com/articles/nmeth.2259 [1] https://www.biorxiv.org/content/10.1101/2021.08.13.456309v1 https://www.biorxiv.org/content/10.1101/2021.08.13.456309v1
- 5y ago
- nickysielicki 5y agoI think the problem with this is that the AI is being used on metrics and images and tests that have meaning to humans. But if we started to take diagnostics that had higher dimensions and resolution, and just trained blackbox AI on that data, it probably stands to reason that it could do a better job. Especially with a tight feedback loop that would identify interesting regions and rescan them instantly, instead of having to bring patients back into the room at a later date. It's the same problem I have with self-driving cars. We are teaching cars to behave around (and like) humans, instead of just fencing off the highways and inventing an autonomous system of vehicles that doesn't have this much tougher constraint. I think AI can do a lot of good things today, if the people trying to apply it looked to completely redesign existing systems around AI, instead of trying to replace the humans operating the existing systems. The latter is much more difficult.
- Swizec 5y ago> instead of just fencing off the highways and inventing an autonomous system of vehicles As a popular joke likes to say: The best place for AI cars is on special roads. You can then make those cars drive very close togehter, even touching. Oh and you can then move the special road behind buildings instead of the front. Give room for more pedestrian friendly pleasant streets. Then you may as well move the road underground. You have invented the subway.
- Zababa 5y agoWe have automated subway where I live, and while they're nice and modern, I understand why people would want the privacy and comfort of a personal car. That's one of the points that isn't mentionned in your joke. Another is that you could have your car use its AI in the special roads in the city and highways, and the rest of the time drive yourself, where AI is harder to implement. A bit like some hybrid cars where you can do most of the day to day commute on electricity, but for longer road trips you can use gas.
- cratermoon 5y agoSort of. I think there's a good argument to be made for part-time self-driving cars with specially-designed highways in places where it makes sense. So you'd drive your car the old-fashioned way from your home on the residential streets until you get to the Automatic Interstate. There, you maneuver into the "pattern" lane and the road and the AI pull you into traffic and drive you to your exit. At that point you're directed out of the pattern and control is given back to you, where you take the side streets to wherever you were going. It has the advantage that it covers the very common use case where someone says they can't take public transportation because they need mobility at their destination.
- honksillet 5y agoSome background info on mammograms (x-rays of the breast). For whatever reason breast cancer is highly political in the US. As a result there are federal laws regarding communication of results and scheduling follow-up exams that apply to mammograms only and no other radilody or other medical test. To comply with these laws radiologists set up a uniform grading system with highly specific jargon. As a result the formatting and language used in mammogram reads by radiologist will be much more consistent than, say, a chest x-ray screening for lung cancer. This makes mammograms the ideal studies to be tackled by AI
- Calloutman 5y agoI'm puzzled at how breast cancer could be a political issue. How so?
- kaesar14 5y agoAmericans have a weird relationship with breasts. Also issues that only affect women (mostly, I know men can get it in rare cases?)
- ars 5y agoBreast cancer is the cancer that raises the most money (that pink ribbon you see everywhere). That money goes somewhere..... This is not a "weird relationship" thing, rather women got organized and: https://en.wikipedia.org/wiki/Breast_cancer_awareness https://en.wikipedia.org/wiki/Breast_cancer_awareness
- make3 5y agoYou have to add a context where the number of radiologist is extremely limited, otherwise the exercise is pointless, but: a different question I would have is, do the AI models know when they are unsure. If so, you could use the AI, and keep the prediction when it is certain, and call a radiologist otherwise. Obviously, in this scenario, you have measured the accuracy of the certainty prediction.
- mcguire 5y agoThat's the neat thing about statistical AI techniques: you give it a training set with two conclusions, yes and no. It will then give you results of yes or no, with absolute certainty. For example, check the photo-recognition systems where minor changes to the picture change the recognition from a parrot to a Ferrari.
- EGreg 5y agoSo the other 6% were more accurate? And the top 2%?
- somenewaccount1 5y agoonly more accurate than a single radiologist review. Current practice is 2 radiologists review. 0% were more accurate when compared to the current practice.
- deleted 5y ago[deleted]
- RamblingCTO 5y agoNothing surprising here for me to be honest. Data is almost always misrepresented, even in AI research. I haven't seen many papers showing the full distribution of result measures, only showing statistics about those measures, which can be disguising what's really going on (also made worse by implications of distributions which are not correct). My AI days have been over for a few years now since I finished my masters, but until then I *never* could reproduce anything published in papers, even using official packages. Data selection bias was (is?) a real problem.
- spoonjim 5y agoWe’re not there yet. But the AI systems are getting better a lot faster than the humans are.
- spywaregorilla 5y agoWhat's up with figure 4A having the human results at 100% sensitivity, 0% false positive rate? What is this trying to imply?
- cratermoon 5y agoThat they compared the results with an independent test (in this case, biopsies) that isn't subject to the same statistical errors?
- spywaregorilla 5y agoIs that it? I don't think so. That wouldn't make any sense. It seems to me it's trying to suggest the radiologists had 100% prediction accuracy. But that can't possibly be true.
- voldemort1968 5y agoPut another way, 2 of the 36 AI systems evaluated were more/as accurate than a single radiologist.
- cratermoon 5y agoMaybe? Or maybe they just got lucky.
- m0zg 5y agoHuh? 6% of the systems outdoing a human who spent years training for her profession is supposed to be bad news? That's awesome! Especially considering that these are not supposed to replace radiologists (yet), but merely to augment expert judgment.
- cratermoon 5y agoThat's assuming they can consistently do it, that the results in the paper were not just random luck, and not taking into account the costs and risks associated with inaccurate diagnoses.
- m0zg 5y agoSure, but this is all _very_ early. You should see the kind of AI "techniques" some of those papers are using. It's not Imagenet, but people just take "academic" models from papers (sometimes from prehistoric times, like 7-10 years ago) and try their luck with them. News flash: model architectures and training regimes are overfitted quite severely to academic datasets such as Imagenet, OpenImages and MSCOCO at this point. When you see a bear riding a bicycle, you don't complain that the bear rides poorly. You're in awe that it can ride at all. Wait until people who know what they're doing move in. Right now a lot of those people are in academia, trying to squeeze the last 0.01% of lift out of Imagenet dog breeds via architecture searches at $100K a pop.
- deleted 5y ago[deleted]
- mattlutze 5y agoThe imprecision of this summary title is like a metaphor for the mixed messages and real-world performance of the systems studied.
- cratermoon 5y agoHow would you have summarized the title to fit HN limits?
- calvinmorrison 5y ago94% of 36 is 33.84
- danieldrehmer 5y agoHumans are underrated
- davnn 5y agoIt's a pity that they only combined two human models; would be interesting to combine some of the AI systems and check again.
- megaman821 5y agoIs this a thing that AI can't be good at, or a thing where the training data set is too small? I can imagine clearing all the privacy and legal hurdles to get 100 million annotated mammograms to be quite the feat, and most AI have to train on very little data.
- sidlls 5y agoHIPAA compliance has many loopholes. Providers and carriers (insurance) are bound by them, but random private companies are not generally. Most covered entities are going to insist anyone they share data with also be compliant, but it's not necessarily a requirement. Not to say it's trivial, either, but it's not exactly a huge challenge to get 100M annotated anything, really.
- 256lie 5y agoThere are efforts like the UK Biobank but healthcare institutions are very sensitive about their patient data.
- cratermoon 5y agoHIPAA is about not identifying the patient. It doesn't say you can't take 100M mammograms, wipe out the PII, and add "this image is of a patient that disease X". I'm not sure how much other demographically interesting metadata you can add, like age, gender, height/weight, etc.
- n8henrie 5y agoI wonder how long it took for the written interpretation in each case? I work in a rural ER where the "stat" reads often take about an hour, sometimes up to 5 hours, and the quality is... such that I've gotten a lot of practice at doing my own reads. Something that is "not as good as a radiologist but maybe better than me," and with results in minutes instead of hours, would be HUGE, if even to just prompt me to review the images myself before (instead of after) seeing the next patient.
- 16bytes 5y agoAs others here have pointed out, the goal shouldn't be to replace radiologists, but to enhance their capabilities. That is to say, pair human judgement with AI. It may not be as good as two humans making a judgement call, but it's probably better than just one. Furthermore with AI you could do this on 100% of exams, as compared to the small amount that go through peer review.
- 256lie 5y agoWould be interesting to see the time advantage. Mammography is high-volume and low-prevalence task with standards such as BI-RADS. While AI will not replace radiologists, breast cancer screening is a prime application to assist with radiologist burnout. I believe Europe already approved AI to be used as a second reader.
- aaron695 5y agoThe most telling is this is 100% known. Yet the HN comment fantasy land you can read above is mostly making excuses or not even aware if hospitals use AI (or not, as is the case) to save lives. The lack of technological knowledge base here is extreme. Copilot, a far more complex task, is going to help programmers code next. Reality is very broken in the IT community.
- joebob42 5y agoI see consistent mention of "2 radiologists review". Is there some feasible way to combine two of these models into some sort of metamodel and get a similar benefit? Like is two radiologist review a mechanical way of combining the opinions of two radiologists (in which case let's just do that with the models), or is it that two radiologists actually sit down together and look at the thing which you can't necessary replicate by just using two models.
- rcarmo 5y agoThere are two sides to this: enhance human decision (by accelerating it) or try replacing it. I played around with a number of image segmentation models a couple of years back and I can only imagine how much more subtle densitometry stuff can be, but overall I'm not surprised accuracy can be way off because _any model_ can be way off depending on what it's presented with. Most commercial Machine Learning doesn't actually learn on the job, and is actually positioned for the triage/faster handling scenario. It will spot what it was trained for, but _most likely only in the conditions it was trained for_. Change the angle, change the patient's age, add any other visible medical conditions, etc., and ratios are likely to drop. Then again, maybe the study could be a little more systematic (as other comments point out). I'd like to see larger numbers, if only because it would help statistic significance.
- miket 5y agohttps://twitter.com/andrewyng/status/930938692310482944 https://twitter.com/andrewyng/status/930938692310482944
- gojomo 5y agoSo, pick the 2 (out-of-36) that did better than a solo radiologist & run them everywhere, at nearly zero marginal cost. Also check if ensembles of the lesser-performing systems (with other systems or with radiologists) can do better, and whether any of the systems can rapid-classify 'easy cases' so expensive radiologists need only check the tougher ones.
- sfink 5y agoIt would be interesting to compare the results of AI + 1 radiologist to 2 radiologists (and with a larger sample, to AI + 2 radiologists). Does the AI add a useful perspective? Does the AI perspective add enough that it's a better backup check than a radiologist? If not, does it add anything to a pair of radiologists, since it would presumably be fast and cheap and so it would still be a good outcome to add it into the process. Another interesting question, if you had unlimited resources for studies: how does AI compare to 1 or 2 poorly trained 3rd world radiologists?
- 256lie 5y agoFor screening, it depends on the false positives rate. A radiologist with have to check every positive prediction. Although, I believe in Europe, they have approved AI to be used as a second reader.
- Jonanin 5y agoThis implies that there were a couple of AI systems that actually beat a radiologist, which I take as extremely promising for the field of AI radiology. Like any domain in applied AI, there will be a lot of approaches that miss the mark, or are simply stepping stones to better approaches. There are thousands and thousands of papers on language modeling, but we only needed one superior approach (GPT) to change the game entirely. The search through any cutting edge problem space is messy and full of failure, and that's fine. You only need one breakthrough.
- okl 5y agoYou could also say: There exist at least some radiologists that can be beaten by an AI system. :D I guess a good questions could be: Will they be more reliable than radiologists in scenarios other than the ones studied?
- rafaelturk 5y agoTks for this wellwriten, fact driven comment.
- arcticbull 5y agoI'd push back on the 6% of AI systems being better than a radiologist and calling that a success, but you are right in the meta. It's fair to say that yes, AI systems aren't good enough yet. On the other hand, it's pretty clear some technological approach will outperform a radiologist at pattern recognition at some point in the future - whether that's "AI" or "if statements" or some third option. It's just a matter of time.
- 1024core 5y ago> I'd push back on the 6% of AI systems being better than a radiologist and calling that a success, How much time does it take to train a radiologist to that level of performance? How much time does it take to clone that ML model?
- arcticbull 5y ago
- TrackerFF 5y agoWhat about the ensemble of multiple AI systems, compared to a single radiologist? In my mind, many AI systems will still be faster than a single radiologist (though this assumption is a completely random guess from my part, I don't know how fast these analysis robots can work.)
- WaitWaitWha 5y ago> 94% of 36 AI systems evaluated were less accurate than a single radiologist Today 94% of 36 AI systems evaluated were less accurate than a single radiologist. FTFY In the beginning of the eDiscovery industry was here too. Lawyers read through insane amount of PDFs, TIFFs, emails and they were better than the AI/ML software presented. Today most courts will reject human eD if AI/ML was available, and no proper legal team will use lawyers to search through terabytes of data 'manually'. The false negatives/positives are significantly better with AI/ML.
- cratermoon 5y agoApples to Oranges. What are the risks of poor discovery for the clients compared to the risks to a patient with a fatal disease mis-diagnosed?
- WaitWaitWha 5y agoIn civil cases millions to billions of <insert your currency> settlements. In criminal cases, decades in jail, execution.
- OneEyedRobot 5y agoI admit that radiology sounds like a tractable problem for machine learning, they'll probably get better. It strikes me, without any real knowledge but this is the internet, that it just goes down the typical rabbit hole of human specialization. Make the data portable, move the data handling overseas, add computer optimization over time. You do have to wonder if radiologists (any here?) really need general medical knowledge or if high specialized training with pictures would be enough.
- 908B64B197 5y agoStage 0: "A computer will never beat a human at chess, the game is simply too complex" Stage 1: "So one managed to do it, but 96% of implementations still can't" Stage 2: "Well, looks like chess is pretty much dominated by computers. But the Go Game is still way too complex for a mere machine to tackle: only the human mind can" Stage 3: "It was one tournament that AlphaGo won and really..." Stage 4: "Yeah, the AI can play better Go and Chess than humans. These are solved problems." I guess we're at stage 1 for self driving and apparently for mammograms as well. Only difference is, chess players didn't have a legalized cartel working for them in the 90's.
- xyzelement 5y agoI actually have a bet with a radiologist that AI will be as good or better than radiologists at reading images in a fairly near future. The reason I think this (as a non-specialist) is this: when I upload a random photo to Google or Facebook, those systems recognize the faces of people in those pictures without any prompt. This seems like a much more constrained problem (given a pretty specific set of images of chest cavities, state the probability of a cancerous tumor.) I am guessing that there's nothing inherent in this problem that is much more difficult than other image recognition problems. I suspect that this outcome is because we have fairly new technology competing against highly specialized humans doing a very specific task and doing it well. I suspect that will happen is that the technology will relatively quickly catch up to humans and do equally well, after which point it's just a speed of response and economics question, where the technology wins over the person.
- greazy 5y agoI'll take that bet against you. > I am guessing that there's nothing inherent in this problem that is much more difficult than other image recognition problems. This is factually wrong. You're assumption I guess is that its a data issue: if we can just get more data we'd solve this issue. The reality is that these diseases are complex, their presentation is complex, and compounded by different technology. There's also the issue of data complexity, x-rays contain less information compared to a picture of your dog. We need better algos, not more data.
- colinmhayes 5y agoHuman faces are complex too. Yet computer vision has no trouble recognizing you out of the billions that exist.
- greazy 5y agoYou've misunderstood my point I think. How a disease presents is different issue than the complexity of faces. A photo of a face has more information encoded than an x-ray image. This lack of information is part of the problem that I think more data doesn't solve and why humans are able to identify cancer. We have better image recognition 'software'. And that's my point, better algos, not more data
- YeGoblynQueenne 5y ago>> Studies were of poor methodological quality. That's the main conclusion to draw. Better studies are needed until we can know whether Geoff Hinton was right [1]. ____________ [1] “Machine learning pioneer Geoffrey Hinton said, ‘If you work as a radiologist, you are like Wile E. Coyote in the cartoon; you’re already over the edge of the cliff, but you haven’t looked down.’ “Hinton went so far as to recommend that med schools stop training radiologists right now.” https://blionline.org/are-cpas-like-wile-e-coyote-off-the-cliff-without-looking-down/ https://blionline.org/are-cpas-like-wile-e-coyote-off-the-cl...
- qwertox 5y agoToday I got my chest x-rayed to search for a broken rib. When we looked at the x-rays, I was wondering how the hell someone could recognize anything there. The doctor didn't find anything and just said if it still hurts in 3 weeks, we'll do another one, maybe we missed it. Same last year, when I clearly broke a rib: first x-rays revealed nothing, one week later a second set revealed a really nice oblique displaced fracture, which was still hard to find. Today the tibia also got x-rayed and it was a pleasure to look at, but as soon as it has to do with the chest, where all the organs are messing with the x-ray, one can really hope that AI will finally solve this issue, since every city has a couple of doctors with x-ray machines, contrary to MRI/CTs.
- jcims 5y agoCome on, most of us are just sitting around burnishing our fingertips on a small piece of glass. Any more details on how you wound up looking at an xray of your ribs twice in ~1 year?
- qwertox 5y agoI started mountain biking around 2 years ago. I think my front suspension is too soft for my weight, so both times the front wheel got in a trough(?) and the suspension gave in too much, making me fly over the handlebar. The tibia was because the bike landed on my leg after flying over me. I'm reconsidering my life choices regarding MTB, since I'm already in my 40s and made zero exercise for at least 20 years. But this time I learned. I will never risk it in the summer, since those sunny days are way to valuable for spending the time on the bike instead of recovering. I told me this last year, and this time it definitely sunk in. "Be careful, you're not a teenager". But it makes so much fun. It is so good to breathe in that fresh air and to exercise hard, I always end up with a smile. Too bad I didn't start with this 20 years ago. I had the chance.
- jcims 5y agoThank you!!! I'm in your same boat, maybe worse b/c I'm deep into my 40's, but just never really did much outdoor activity. When I was a kid there was all sorts of federal lands around that we could really roll around in however we liked. Where i'm at now it's a 40 minute drive to trails, so it's just streets and other people's property. :/ Just now getting set up to get more time outside . Sounds like i should take a lesson from your pain. :) Hope you get feeling better soon and get back out there (if maybe a tick lower kinetic energy :) (also i meant to put a smiley in my first reply, sorry for coming across as a dick)
- CultMember002 5y agoWe should not allow the Fed to stifle innovation and technological progress. If we are not willing to take risks, then we will never be able to reach out true potential of becoming a truly advanced space faring civilization and a multi-planetary species. The NHTSA does not have the technological capability to understand advanced AI used in Autopilot and Full Self Driving and they should stop their interference immediately. I have a Tesla and I bought the Full Self Driving package. I believe it is far more valuable than the current price. I am willing to pay 30 times more for Full Self Driving.
- deleted 5y ago[deleted]
- greatacrep 5y agoBrigade Group Contact Information-Brigade Komarla HeightsApartments-Brigade Group Address-Brigade Group Bangalore-Brigade Komarla Heights-Apartments For sale-Luxury Apartments For sale-1 BHK Apartments-2 BHK Apartments-3 BHK Apartments -Brigade Komarla HeightsAbout Us-Brigade Komarla HeightsOverview Visit : https://www.brigadekomarlaheights.org.in/ https://www.brigadekomarlaheights.org.in/
- ZeroGravitas 5y agoI've mentioned before that I track technological progress by the change in criticisms. It appears that AI screening of breast cancer has reached the "not always better than a human expert" stage which is pretty good, that would be sci-fi like 10 years ago. Makes me wonder how a human-AI "centaur" or the averaged output of multiple independent AIs would do, since the standard already seems to be multiple humans to catch everything.
- rscho 5y ago'I am a software engineer and I think general AI is upon us but hindered by nasty doctors who want to get rich, therefore I will make every effort to believe that this paper shows that AI is better'
- aabajian 5y agoI'm a 5th year radiology resident with undergraduate and master's degrees in computer science...and currently on a mammography rotation. A couple background facts: 1. Mammographer subspecialist salaries are now the highest of any radiologist (median ~$440K) - https://www.auntminnie.com/index.aspx?sec=ser&sub=def&pag=dis&ItemID=128314 https://www.auntminnie.com/index.aspx?sec=ser&sub=def&pag=di... 2. Mammography is one of the easier fellowships to undertake and is "lifestyle-friendly" overall (no call as an attending!) 3. Mammographers are highly in-demand, even in desirable locations. So it's (relatively) easy, very well-paid, and you can live where you want. What's the catch? Well... 1. Mammographers are the most likely to be sued. Missing a mammographic diagnosis of breast cancer is the #2 reason for medical litigation, with some sources citing up to 10% of radiologists per year being named in a mammography litigation case. See: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3138733/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3138733/ and https://pubmed.ncbi.nlm.nih.gov/22596034/ https://pubmed.ncbi.nlm.nih.gov/22596034/ 2. Mammographers directly interact with patients to relay findings and biopsy lesions, and they must collaborate with referring providers and pathologists. Breast cancer is a highly emotional topic in the United States due to a combination of public personas ('Angelina Jolie effect') and political maneuvering (see the history of the federal Mammography Quality Standards Act and Program - MQSA). You can't be a "behind the scenes, in the dark" radiologist if you go into mammography. From a computer science perspective, I one-hundred-percent agree that mammography is a "solvable" problem by AI. Standardization of mammography images (CC and MLO screening views) alongside BI-RADS makes for an amazingly well-labeled dataset. I fully expect AI to surpass the abilities of one-or-more mammographers in the near future. And yet, amongst radiologists, mammographers are the least likely to be replaced. Why? For the same reason mammography is lucrative to begin with - Litigation, emotional patient-doctor interactions, and the social/political climate. Any one of these is a steep hill by itself. Most AI companies want none of the legal risk, and practically zero of them offer patient-facing results. And, I wonder if any of the AI startups have considered hiring a celebrity as their spokeperson. Perhaps politics/legislation might work in favor of AI if it becomes standard of care, but I wouldn't want to bank my startup on a federal regulatory change.
- naveen99 5y agoHey, congrats on nearing the end of formal training. The other issue with Mammography is that even human accuracy isn’t great. which is why they keep getting sued. So what specialty did you chose ? Call me if you are interested in body / oncological imaging.
- tpoacher 5y agoConclusion: Radiologist manual segmentations are 94% useless when generated as ground truth, and the AI community needs to find an alternative, fast.
- dontreact 5y agoI quickly read through some of the claims that the articles make about the studies. While I agree wholeheartedly that there are a lot of false claims about medical AI swirling about, I don’t trust the authors of this paper to tell us where it’s happening based on the mistakes I’ve seen here “ The remaining studies used enrichment leading to breast cancer prevalence (ranging from 7.4%26 to 73.8%37), which is atypical of screening populations. Five studies used reading under “laboratory” conditions at risk of introducing bias because radiologists read mammograms differently in a retrospective laboratory experiment than in clinical practice. Only one of the studies used a prespecified test threshold which was internal to the AI system to classify mammographic images.” I’m very close to the authors of McKinney et. al and I happen to know that 2/3 of these claims are either false or specious: 1. The operating point was decided ahead of time before the reader study that was conducted in the paper. I was literally there and saw Scott choose it and I saw his methodology for doing so. So this claim that they did not use a prespecified threshold I have first hand evidence of it being false. 2. Enrichment: Enriching for positives is absolutely a standard practice and it has mathematically 0 impact on computed metrics. It’s not possible to conduct a reader study without enrichment because cancer is so rare that you would never be able to recruit readers to your study. It just doesn’t make sense at this stage of research to not do enrichment because regardless of what you do the study will still be retrospective. The paper also shows that results on the unenriched data is basically the same. I think the understanding that we had at Google and that was mentioned in the paper is that these types of retrospective studies are not sufficient to deploy the AI. It’s probably a good thing this paper is stressing those points. We need prospective studies, but practically speaking it makes sense to first do really solid retrospective studies. Otherwise a hospital system is not going to just let you run your AI on their scans. Google is proceeding to do this sort of testing with the work that this paper is just falsely thrashing: https://news.northwestern.edu/stories/2021/02/artificial-intelligence-breast-cancer/ https://news.northwestern.edu/stories/2021/02/artificial-int... I feel that this article has good intent (stressing the importance of prospective trials rather than just retrospective studies) However, the execution is significantly flawed and the outcome seems to be in this particular forum people believing that AI will never work for this application. These things take time and there is a pipeline of testing and trials that takes many years to go through. 1. Research and development 2. Retrospective testing: as a replacement system (so you don’t have to figure out the UX of how to make it help the human) 3. Retrospective testing: as a helping system (generally scientific papers have been less interested in this step and it would be nice to see more interest here) 4. Prospective testing as an assistive system. I think the Google system is on step 4 now, and this paper is looking at evidence published from stage 2 and claiming the system does not work. It’s probably a fair claim because it has not completed all the testing to be deployed. There is as of yet no intent to deploy it as a replacement system, but rather as some sort of assistive system so that prospective evidence can be collected. However the way they’ve done this is by misrepresenting the paper, and I really think they’ve overstated their case here.