6 ms·

AI misses nearly one-third of breast cancers, study finds

153 points by Liquidity 9mo ago

andrewstuart 9mo ago
AI finds nearly 2/3rds of breast cancers!
- thereisnospork 9mo ago
  Depending how the costs of AI detection vs doctor, that genuinely might be enough to shift the math and be a net positive. If it is cheap enough to test 10x the current tested population, which would have lower, but non-zero rates of breast cancer, then[0] AI would result in more cancer detected and therefore more aggregate lives saved. [0]presumptively
  - ludston 9mo ago
    Given that every positive case needs to be verified by a doctor anyway because the patient has breast cancer, and every negative case has to be checked because it does a worse job than traditional methods... It only costs more.
    - thereisnospork 9mo ago
      Depends on the false positive rate. Hypothetically one can 'just' tune the model so false positives are low. This will increase false negatives but those are 'free' as they don't require follow ups. So long as the decrease in cost per real positive[0] goes down there's a benefit to be had. [0] accounting for false positives, screening costs for true negatives, etc. etc.
      - friendzis 9mo ago
        > This will increase false negatives but those are 'free' as they don't require follow ups. Increase in false negative rate significantly reduces survival rate and increases cost of treatment. We have huge multiplication factor here so decreasing false negative rate is the net positive option at relatively low rates.
  - moooo99 9mo ago
    > Depending how the costs of AI detection vs doctor, that genuinely might be enough to shift the math and be a net positive. Based on my very superficial medical understanding, screening is already the cheap part. But every false-positive would lead to a doctor follow up at best and a biopsy at worst. Not to mention the significant psychological effects this has on a patient. So I would counter that the potential increase of false-positive MRI scans could be enough to tip off the scale to make screening less useful
- nomel 9mo ago
  I see you're not in the marketing department! We can do better by only considering "missed" as what wouldn't also be missed by a human: AI finds 71% of breast cancer*! *Compared to a human.
emil-lp 9mo ago
Please always present the confusion matrix. One number is (almost) useless. I can detect 100% by def detect(x): Return True
- rich_sasha 9mo ago
  This fails actually (assuming it's Python), "return" needs smaller case "r". But you could rewrite it in Haskell or Rust for safety.
- dtech 9mo ago
  They only had positive samples, so they could and did only report true and false positives
  - JohnKemeny 9mo ago
    While they only had positive samples, the AIs sometimes reported cancer in the wrong location, meaning a double whammy: It failed to detect the cancer, and misdiagnosed a non-cancer. From the paper: > Two cancers had abnormality scores greater than 10 but were not correctly localized and were therefore categorized as AI-missed.
- patates 9mo ago
  100% of the titles are not specific enough on sensitive matters.
- directevolve 9mo ago
  There were no healthy controls, so they can only measure sensitivity, not specificity in this design.
levocardia 9mo ago
The original study: https://link.springer.com/article/10.1007/s11547-025-02161-1 https://link.springer.com/article/10.1007/s11547-025-02161-1 It was retrospective-only, i.e. a case series on women who were known to have breast cancer, so there were zero false negatives and zero true negatives, because all patients in the study truly had cancer. The AI system used was a ConvNet used commercially circa 2021, which is when the data for this case series were collected.
- boxed 9mo ago
  I believe there's a big issue in the US of over-diagnosing breast cancer too. "Known to have breast cancer" might not be so clear cut.
  - ako 9mo ago
    That statement would benefit from a link to your source.
    - boxed 9mo ago
      https://news.cancerresearchuk.org/2025/11/18/overdiagnosis-when-finding-cancer-can-do-more-harm-than-good/ https://news.cancerresearchuk.org/2025/11/18/overdiagnosis-w... Is about the UK, but it's a good description of the danger of over-diagnosis. When the treatment for a diagnosis includes radiation therapy, over-diagnosis can literally cause cancer.
- operatingthetan 9mo ago
  >The AI system used was a ConvNet used commercially circa 2021, which is when the data for this case series were collected. Does this mean that newer AI systems would perform significantly differently?
  - dfee 9mo ago
    Well, certainly not. We shouldn’t draw conclusions about modern AI systems from multi-generation old systems: one way or the other.
Moosturm 9mo ago
Shouldn't A.I. not be used in a way that it only tries to assist? E.g. a doctor takes a look first and if (s)he can't find anything then A.I. is checking as well (or in parallel).
- f1shy 9mo ago
  My personal opinion: AI should be still kept out of anything mission critical, in all stages, except for evaluation. There is other comment very correctly noting that this result is on 100% positive input. Same AI in “real life” would score probably much better eventually. But as you point out, if used as a confirmation tool, is definitely bad.
  - friendzis 9mo ago
    > Same AI in “real life” would score probably much better eventually Either I don't understand your reasoning or you are very much wrong. A "real life" dataset would contain real negatives too and the result would be equal if false positive rate was zero and strictly worse if the rate was any higher. One should expect the same AI to score significantly worse in a real life setting.
    - f1shy 9mo ago
      Depends on what you call better or worse. In real life positives (TN) are far less common than negatives (TN), if this system does not have lots of FP (which is very possible), the accuracy will be much better than you may expect. What I mean with “score” is having a relatively high accuracy. Come let’s do the math: incidence of BC is 1 every 12, lets say. Now let’s say we have 12000 patients: Acuracy = (TP + TN) / (TP + TN + FP + FN) = (1000 + 11000) / (1000 + 11000 + 300 + 0) = 12000 / 12300 =0.976 the test is 97.6% accurate… pretty impressive huh? Tell me if I’m wrong. Is a know fact that you have to be careful when doctor speak of % accuracy.
- directevolve 9mo ago
  In case you missed it, they weren't using AI to make these diagnoses.
docdeek 9mo ago
In the human follow-up, there was an improvement but there was still a gap: > Their findings offered reassurance: DWI alone identified the majority of cancers the AI had overlooked, detecting 83.5% of missed lesions for one radiologist and 79.5% for the other. The combination of AI and this DWI methodology seems to identify most of the cancer, but there’s still about 20% of 1/3 that gets missed. I assume that as these were confirmed diagnoses, they were caught with another method beyond DWI.
ggm 9mo ago
Useful to show the failure rate for humans, and humans assisted by systems.
- directevolve 9mo ago
  No, this design is not capable of showing this. It did not compare to unblinded humans, did not provide AI with the same data used to make the initial diagnosis, and did not include healthy controls. It measures sensivitity only.
sfink 9mo ago
The title bothers me. It suggests to me that "AI" is a single thing. If two guys are tested and turn out to be not that great at reading MRI images, should the headline be "Male radiologists miss nearly one-third of breast cancers"? If it said "AI something", I'd be fine with it. It's a statement about that something, not about AI in general. Use it as an adjective (short for "AI-using" I guess?), not a noun.
- melagonster 9mo ago
  They will directly write that "Radiologists miss nearly one-third of breast cancers." I trust the meaning of this article is just that it requires hospitals to rethink their decision to substitute all doctors today.
  - philipallstar 9mo ago
    No hospital is deciding that. People have been testing whether we can replace radiologists with AI for over 10 years.
  - pcthrowaway 9mo ago
    If they test someone with no background in radiology, they could even make the headline "Humans miss 50% of breast cancers"
    - literalAardvark 9mo ago
      I wouldn't miss any of them. "Idk what I'm looking at but click positive and let someone who does sort it out this is way too dangerous". AI doesn't have that option yet.
- nottorp 9mo ago
  > It suggests to me that "AI" is a single thing. But it is. It's LLMs. There is no other "AI". Haven't you read HN in the past 1-2 years?
- triceratops 9mo ago
directevolve 9mo ago
The point of this study is that it suggests that fully AI-automated mammography can currently deliver 70% sensitivity in detecting breast cancer using this model. It does not enable us to compare AI to unaided human performance. As this study did not include healthy controls, there is no false positive rate. The false positive rate is a crucial missing metric, since the vast majority of women do not have breast cancer. In nearly half the false negatives from both the mammogram and DWI datasets, the cancer was categorized as occult by two breast radiologists, meaning the cancer was invisible to a trained eye. The AI model's non-occult false negative rate on the mammography data is 19.3%. For that 19.3% figure, see Table 2: 68 non-occult in AI-missed cancer, 285 non-occult in AI-detected cancer. This study did not compare the AI to a radiologist on a mixed set of healthy and cancer images.
- _heimdall 9mo ago
  Its interesting to see this valid argument raised against this use of AI to identify breast cancer. The lack of control groups is one of the more common concerns raised related to vaccines as well, the argument lands like a lead balloon there.
  - dgacmu 9mo ago
    Because it's generally unethical to not give someone a treatment known already to be safe and effective. Studies of new vaccines where there is not an existing vaccine _do_ use placebo controls. Heck, my son got placebo during moderna's pediatric covid vaccine trial (to our frustration. grin.) Subsequent trials generally compare against the best known current treatment as the control instead. This study has no such concerns. It's ethical to include images of non-cancerous breast tissue. The things are not comparable.
    - onetokeoverthe 9mo ago
      [dead]
    - _heimdall 9mo ago
      The covid vaccines were a whole different beast, though interesting case studies they were done under emergency authorization and didn't follow standard protocols. Vaccine studies today almost always use a previously approved vaccine as the "control" group. That isn't a true control and if you walk back the chain of approvals you'd be hard pressed to find a starting point that did use proper control groups. Anyway, my point here wasn't to directly debate vaccines themselves, only to point out that its interest to me as someone without a career in health to see the same effective argument used in two different scenarios with drastically different common responses.
swisniewski 9mo ago
The description from the summaries sound very flawed. 1. They only tested 2 Radiologists. And they compared it to one model. Thus the results don’t say anything about how Radiologists in general perform against AI in general. The most generous thing the study can say is that 2 Radiologists outperformed a particular model. 2. The Radiologists were only given one type of image, and then only for those patients that were missed by the AI. The summaries don’t say if the test was blind. The study has 3 authors, all of which appear to be Radiologists, and it mentions 2 Radiologists looked at the ai-missed scans. This raises questions about whether the test was blind or not. Giving humans data they know are true positives and saying “find the evidence the AI missed” is very different from giving an AI model also trained to reduce false positives a classification task. Humans are very capable at finding patterns (even if they don’t exist) when they want to find a pattern. Even if the study was blind initially, trained humans doctors would likely quickly notice that the data they are analyzing is skewed. Even if they didn’t notice, humans are highly susceptible to anchoring bias. Anchoring bias is a cognitive bias where individuals rely too heavily on the first piece of information they receive (the "anchor") when making subsequent judgments or decisions. They skewed nature or the data has a high potential to amplify any anchoring bias. If the experiment had controls, any measurement error resulting from human estimation errors could potentially cancel out (a large random sample of either images or doctors should be expected to have the same estimation errors in each group). But there were no controls at all in the experiment, and the sample size was very small. So the influence of estimation biases on the result could be huge. From what I can read in the summary, these results don’t seem reliable. Am I missing something?
- directevolve 9mo ago
  The did NOT test radiologists. There were NO healthy controls. They evaluated AI false negative rate and used exclusively unblinded radiologists to grade the level of visibility and other features of the cancer. Utility of the study is to evaluate potential AI sensitivity if used for mass fully automated screenings using mammography data. But says NOTHING about the CRUCIAL false positive rate (no healthy controls) and NOTHING about AI vs. human performance. See my main comment elsewhere in this threat.
joelthelion 9mo ago
"AI" doesn't exist. There are probably hundreds of different breast cancer detection algorithms. Maybe the SOTA isn't good enough yet. That doesn't mean AI in general is fundamentally incapable of correctly detecting it.
- nephihaha 9mo ago
  It is a useful umbrella term for all the systems in question.
  - joelthelion 9mo ago
    It seems this study tested a single one of them.
    - nephihaha 9mo ago
      These reports often misrepresent scientific data, I think that's a given.
noobermin 9mo ago
The moving of the goal posts is a lot here. Measuring sensitivity only is still a useful thing and at least aids radiologists in their decision to use this model specifically and how to base their reliance on it. Also, why does every study have to compare all humans across human history to some particular model?
Traster 9mo ago
This seems a bit like a needlessly publicized finding. Surely our baseline assumption is that there are lots of systems that aren't very good at finding cancer. We're interested in findings that are good. You only need 1 good system to adopt. Yes, it's good scientific hygiene to do the study and publish it going "Well, this particular thing isn't good let's move on". But my expectation is you just going until you design a system that does do well and then adopt that system. If I pluck a guy off the street, get him to analyze a load of MRI scans and he doesn't correctly identify cancer from them I'm not going to publish an article saying "Humans miss X% of breast cancers" am I.
- ForHackernews 9mo ago
  Many people are confused and think the Bitter Lesson is that you can just feed up a bigger and bigger model and eventually it becomes omnipotent.
  - shevy-java 9mo ago
    They promised us that AI was The Solution. Now they have to deliver. If the TechBros fail us here, we may then assume they may fail us everywhere else as well.
    - lambdaone 9mo ago
      And we'd be wrong about that. Different domains are showing wildly different characteristics, with some ML models showing superhuman or expert level performance in some domains (chess, face and handwriting recognition for example) and promising but as yet just not good enough in other domains (radiography, self-driving cars, question answering, prose writing). Currently coding is somewhere in the middle; superhuman in some ways, disappointingly unusable in others. I don't we can make any conclusive verdict about the promise of ML for radiography right now; the life-critical nature of the application it's in the unusable middle, but it might get better in a few years or it might not. Time will tell.
- Ekaros 9mo ago
  I think finding that AI or at least specific model sold to be able to do something can't reasonable do it is entirely reasonable thing to publish. In the end it is on the model marketer to prove that what they sell can do what it says. And counter examples is fully valid thing to then release.
shevy-java 9mo ago
So basically AI kills people. This is Skynet 2.0 or 3.0. But shit. James Cameron may have to redo The Terminator, to include AI. Then again, who would watch such a movie?
NVHacker 9mo ago
Not all AIs are created equal.
nephihaha 9mo ago
As someone else has pointed out, I would like to know how this compares to humans. I hope something good comes out of this, as I have known women whose lives were deeply affected by this.
davidguetta 9mo ago
This is a terrible article. "One AI is not great" is not an interesting finding and certainly not conclusive of "AI can't help or do the job". It's like saying "some dude can't detect breast cancer" and suggesting all humans are useless.
zqy123007 9mo ago
okay guys, I developed AI mammo screening product, let me clear things up. you read it wrong, and I don't blame you. I doubt whoever wrote this actually have a good understanding of the numbers. the setup: 1. 400s confirmed patients 2. AI reads Mammography ONLY and missed 1/3 3. on those AI missed patients, radiologists do a second read on MRI, which is the gold standard for differential. evidence: the referenced paper at the bottom <Added value of diffusion-weighted imaging in detecting breast cancer missed by artificial intelligence-based mammography.> So, the whole point it (or its reference paper) is trying to make is: Mammography sucks, MRI is much better, which is a KNOWN FACT. Now, let me give you some more background missing from the paper: 1. Why does Mammography suck? well, go google/gpt some images, its essentially X-ray for the breast, which compress 3D volumes into 2D average poole plane, which is infomation lossy. SO, AI or not, the sensitivity is limited by the modality. 2. How bad/good is Mammography AI? I would say 80~85% sensitivity agaist very thorough+experienced radiologist without making unbearable amount of FP, that probably translates to 2/3 sensitivity against real cancer cohert, so the referenced number is about right. 3. Mammography sucks, what's the point? its cheap AND fast, you can probably do walk-in and get interpretation back in hours. Whereas MRI you probably need to schedule 2 weeks ahead if not MORE. For a yearly screening, it works for the majority of polulation. and final pro tip, 1. breast tumor is more pravelent than you think (maybe 1 in 2 by age 70+) 2. most guides recommend women with 45+ do yearly checkup 3. if you have dense breast (basically small and firm), add ultrasound screening to make sure. 4. breast feeding does good for both the mother and child, do that. peace & love
- overtone1000 9mo ago
  Radiologist here, thanks for posting this because I was biting my tongue. I'll supplement by directing others to consider how number needed to screen may be a more useful metric than mammographic sensitivity when making policy decisions. They're related, obviously, but only one of them concerns outcomes.
titaniumrain 9mo ago
The missed cases should be attributed to the specific model deployed in the product, not to AI as a general concept. Framing this limitation under a broad and alarming title is therefore misleading and unnecessary.
samrus 9mo ago
Whats the baseline? How many did a human being get? We need to compare this to baseline to know if its good or bad
- directevolve 9mo ago
  As there were no healthy controls or comparison with blinded radiologist performance, it is not possible to answer those questions using this study. The point was to evaluate AI sensitivity, which is useful. However, without healthy controls, it is not possible to determine specificity, which is a crucial statistic since most women do not have breast cancer.
mannyv 9mo ago
Obviously we need more breasts "for training."

153 points by Liquidity 9mo ago

andrewstuart 9mo ago

AI finds nearly 2/3rds of breast cancers!

thereisnospork 9mo ago
Depending how the costs of AI detection vs doctor, that genuinely might be enough to shift the math and be a net positive. If it is cheap enough to test 10x the current tested population, which would have lower, but non-zero rates of breast cancer, then[0] AI would result in more cancer detected and therefore more aggregate lives saved. [0]presumptively
- ludston 9mo ago
  Given that every positive case needs to be verified by a doctor anyway because the patient has breast cancer, and every negative case has to be checked because it does a worse job than traditional methods... It only costs more.
  - thereisnospork 9mo ago
    Depends on the false positive rate. Hypothetically one can 'just' tune the model so false positives are low. This will increase false negatives but those are 'free' as they don't require follow ups. So long as the decrease in cost per real positive[0] goes down there's a benefit to be had. [0] accounting for false positives, screening costs for true negatives, etc. etc.
    - friendzis 9mo ago
      > This will increase false negatives but those are 'free' as they don't require follow ups. Increase in false negative rate significantly reduces survival rate and increases cost of treatment. We have huge multiplication factor here so decreasing false negative rate is the net positive option at relatively low rates.
- moooo99 9mo ago
  > Depending how the costs of AI detection vs doctor, that genuinely might be enough to shift the math and be a net positive. Based on my very superficial medical understanding, screening is already the cheap part. But every false-positive would lead to a doctor follow up at best and a biopsy at worst. Not to mention the significant psychological effects this has on a patient. So I would counter that the potential increase of false-positive MRI scans could be enough to tip off the scale to make screening less useful
nomel 9mo ago
I see you're not in the marketing department! We can do better by only considering "missed" as what wouldn't also be missed by a human: AI finds 71% of breast cancer*! *Compared to a human.

emil-lp 9mo ago

Please always present the confusion matrix. One number is (almost) useless. I can detect 100% by def detect(x): Return True

rich_sasha 9mo ago
This fails actually (assuming it's Python), "return" needs smaller case "r". But you could rewrite it in Haskell or Rust for safety.
dtech 9mo ago
They only had positive samples, so they could and did only report true and false positives
- JohnKemeny 9mo ago
  While they only had positive samples, the AIs sometimes reported cancer in the wrong location, meaning a double whammy: It failed to detect the cancer, and misdiagnosed a non-cancer. From the paper: > Two cancers had abnormality scores greater than 10 but were not correctly localized and were therefore categorized as AI-missed.
patates 9mo ago
100% of the titles are not specific enough on sensitive matters.
directevolve 9mo ago
There were no healthy controls, so they can only measure sensitivity, not specificity in this design.

levocardia 9mo ago

The original study: https://link.springer.com/article/10.1007/s11547-025-02161-1 https://link.springer.com/article/10.1007/s11547-025-02161-1 It was retrospective-only, i.e. a case series on women who were known to have breast cancer, so there were zero false negatives and zero true negatives, because all patients in the study truly had cancer. The AI system used was a ConvNet used commercially circa 2021, which is when the data for this case series were collected.

boxed 9mo ago
I believe there's a big issue in the US of over-diagnosing breast cancer too. "Known to have breast cancer" might not be so clear cut.
- ako 9mo ago
  That statement would benefit from a link to your source.
  - boxed 9mo ago
    https://news.cancerresearchuk.org/2025/11/18/overdiagnosis-when-finding-cancer-can-do-more-harm-than-good/ https://news.cancerresearchuk.org/2025/11/18/overdiagnosis-w... Is about the UK, but it's a good description of the danger of over-diagnosis. When the treatment for a diagnosis includes radiation therapy, over-diagnosis can literally cause cancer.
operatingthetan 9mo ago
>The AI system used was a ConvNet used commercially circa 2021, which is when the data for this case series were collected. Does this mean that newer AI systems would perform significantly differently?
- dfee 9mo ago
  Well, certainly not. We shouldn’t draw conclusions about modern AI systems from multi-generation old systems: one way or the other.

Moosturm 9mo ago

Shouldn't A.I. not be used in a way that it only tries to assist? E.g. a doctor takes a look first and if (s)he can't find anything then A.I. is checking as well (or in parallel).

f1shy 9mo ago
My personal opinion: AI should be still kept out of anything mission critical, in all stages, except for evaluation. There is other comment very correctly noting that this result is on 100% positive input. Same AI in “real life” would score probably much better eventually. But as you point out, if used as a confirmation tool, is definitely bad.
- friendzis 9mo ago
  > Same AI in “real life” would score probably much better eventually Either I don't understand your reasoning or you are very much wrong. A "real life" dataset would contain real negatives too and the result would be equal if false positive rate was zero and strictly worse if the rate was any higher. One should expect the same AI to score significantly worse in a real life setting.
  - f1shy 9mo ago
    Depends on what you call better or worse. In real life positives (TN) are far less common than negatives (TN), if this system does not have lots of FP (which is very possible), the accuracy will be much better than you may expect. What I mean with “score” is having a relatively high accuracy. Come let’s do the math: incidence of BC is 1 every 12, lets say. Now let’s say we have 12000 patients: Acuracy = (TP + TN) / (TP + TN + FP + FN) = (1000 + 11000) / (1000 + 11000 + 300 + 0) = 12000 / 12300 =0.976 the test is 97.6% accurate… pretty impressive huh? Tell me if I’m wrong. Is a know fact that you have to be careful when doctor speak of % accuracy.
directevolve 9mo ago
In case you missed it, they weren't using AI to make these diagnoses.

docdeek 9mo ago

In the human follow-up, there was an improvement but there was still a gap: > Their findings offered reassurance: DWI alone identified the majority of cancers the AI had overlooked, detecting 83.5% of missed lesions for one radiologist and 79.5% for the other. The combination of AI and this DWI methodology seems to identify most of the cancer, but there’s still about 20% of 1/3 that gets missed. I assume that as these were confirmed diagnoses, they were caught with another method beyond DWI.

ggm 9mo ago

Useful to show the failure rate for humans, and humans assisted by systems.

directevolve 9mo ago
No, this design is not capable of showing this. It did not compare to unblinded humans, did not provide AI with the same data used to make the initial diagnosis, and did not include healthy controls. It measures sensivitity only.

sfink 9mo ago

The title bothers me. It suggests to me that "AI" is a single thing. If two guys are tested and turn out to be not that great at reading MRI images, should the headline be "Male radiologists miss nearly one-third of breast cancers"? If it said "AI something", I'd be fine with it. It's a statement about that something, not about AI in general. Use it as an adjective (short for "AI-using" I guess?), not a noun.

melagonster 9mo ago
They will directly write that "Radiologists miss nearly one-third of breast cancers." I trust the meaning of this article is just that it requires hospitals to rethink their decision to substitute all doctors today.
- philipallstar 9mo ago
  No hospital is deciding that. People have been testing whether we can replace radiologists with AI for over 10 years.
- pcthrowaway 9mo ago
  If they test someone with no background in radiology, they could even make the headline "Humans miss 50% of breast cancers"
  - literalAardvark 9mo ago
    I wouldn't miss any of them. "Idk what I'm looking at but click positive and let someone who does sort it out this is way too dangerous". AI doesn't have that option yet.
nottorp 9mo ago
> It suggests to me that "AI" is a single thing. But it is. It's LLMs. There is no other "AI". Haven't you read HN in the past 1-2 years?
triceratops 9mo ago

directevolve 9mo ago

The point of this study is that it suggests that fully AI-automated mammography can currently deliver 70% sensitivity in detecting breast cancer using this model. It does not enable us to compare AI to unaided human performance. As this study did not include healthy controls, there is no false positive rate. The false positive rate is a crucial missing metric, since the vast majority of women do not have breast cancer. In nearly half the false negatives from both the mammogram and DWI datasets, the cancer was categorized as occult by two breast radiologists, meaning the cancer was invisible to a trained eye. The AI model's non-occult false negative rate on the mammography data is 19.3%. For that 19.3% figure, see Table 2: 68 non-occult in AI-missed cancer, 285 non-occult in AI-detected cancer. This study did not compare the AI to a radiologist on a mixed set of healthy and cancer images.

_heimdall 9mo ago
Its interesting to see this valid argument raised against this use of AI to identify breast cancer. The lack of control groups is one of the more common concerns raised related to vaccines as well, the argument lands like a lead balloon there.
- dgacmu 9mo ago
  Because it's generally unethical to not give someone a treatment known already to be safe and effective. Studies of new vaccines where there is not an existing vaccine _do_ use placebo controls. Heck, my son got placebo during moderna's pediatric covid vaccine trial (to our frustration. grin.) Subsequent trials generally compare against the best known current treatment as the control instead. This study has no such concerns. It's ethical to include images of non-cancerous breast tissue. The things are not comparable.
  - onetokeoverthe 9mo ago
    [dead]
  - _heimdall 9mo ago
    The covid vaccines were a whole different beast, though interesting case studies they were done under emergency authorization and didn't follow standard protocols. Vaccine studies today almost always use a previously approved vaccine as the "control" group. That isn't a true control and if you walk back the chain of approvals you'd be hard pressed to find a starting point that did use proper control groups. Anyway, my point here wasn't to directly debate vaccines themselves, only to point out that its interest to me as someone without a career in health to see the same effective argument used in two different scenarios with drastically different common responses.

swisniewski 9mo ago

The description from the summaries sound very flawed. 1. They only tested 2 Radiologists. And they compared it to one model. Thus the results don’t say anything about how Radiologists in general perform against AI in general. The most generous thing the study can say is that 2 Radiologists outperformed a particular model. 2. The Radiologists were only given one type of image, and then only for those patients that were missed by the AI. The summaries don’t say if the test was blind. The study has 3 authors, all of which appear to be Radiologists, and it mentions 2 Radiologists looked at the ai-missed scans. This raises questions about whether the test was blind or not. Giving humans data they know are true positives and saying “find the evidence the AI missed” is very different from giving an AI model also trained to reduce false positives a classification task. Humans are very capable at finding patterns (even if they don’t exist) when they want to find a pattern. Even if the study was blind initially, trained humans doctors would likely quickly notice that the data they are analyzing is skewed. Even if they didn’t notice, humans are highly susceptible to anchoring bias. Anchoring bias is a cognitive bias where individuals rely too heavily on the first piece of information they receive (the "anchor") when making subsequent judgments or decisions. They skewed nature or the data has a high potential to amplify any anchoring bias. If the experiment had controls, any measurement error resulting from human estimation errors could potentially cancel out (a large random sample of either images or doctors should be expected to have the same estimation errors in each group). But there were no controls at all in the experiment, and the sample size was very small. So the influence of estimation biases on the result could be huge. From what I can read in the summary, these results don’t seem reliable. Am I missing something?

directevolve 9mo ago
The did NOT test radiologists. There were NO healthy controls. They evaluated AI false negative rate and used exclusively unblinded radiologists to grade the level of visibility and other features of the cancer. Utility of the study is to evaluate potential AI sensitivity if used for mass fully automated screenings using mammography data. But says NOTHING about the CRUCIAL false positive rate (no healthy controls) and NOTHING about AI vs. human performance. See my main comment elsewhere in this threat.

joelthelion 9mo ago

"AI" doesn't exist. There are probably hundreds of different breast cancer detection algorithms. Maybe the SOTA isn't good enough yet. That doesn't mean AI in general is fundamentally incapable of correctly detecting it.

nephihaha 9mo ago
It is a useful umbrella term for all the systems in question.
- joelthelion 9mo ago
  It seems this study tested a single one of them.
  - nephihaha 9mo ago
    These reports often misrepresent scientific data, I think that's a given.

noobermin 9mo ago

The moving of the goal posts is a lot here. Measuring sensitivity only is still a useful thing and at least aids radiologists in their decision to use this model specifically and how to base their reliance on it. Also, why does every study have to compare all humans across human history to some particular model?

Traster 9mo ago

This seems a bit like a needlessly publicized finding. Surely our baseline assumption is that there are lots of systems that aren't very good at finding cancer. We're interested in findings that are good. You only need 1 good system to adopt. Yes, it's good scientific hygiene to do the study and publish it going "Well, this particular thing isn't good let's move on". But my expectation is you just going until you design a system that does do well and then adopt that system. If I pluck a guy off the street, get him to analyze a load of MRI scans and he doesn't correctly identify cancer from them I'm not going to publish an article saying "Humans miss X% of breast cancers" am I.

ForHackernews 9mo ago
Many people are confused and think the Bitter Lesson is that you can just feed up a bigger and bigger model and eventually it becomes omnipotent.
- shevy-java 9mo ago
  They promised us that AI was The Solution. Now they have to deliver. If the TechBros fail us here, we may then assume they may fail us everywhere else as well.
  - lambdaone 9mo ago
    And we'd be wrong about that. Different domains are showing wildly different characteristics, with some ML models showing superhuman or expert level performance in some domains (chess, face and handwriting recognition for example) and promising but as yet just not good enough in other domains (radiography, self-driving cars, question answering, prose writing). Currently coding is somewhere in the middle; superhuman in some ways, disappointingly unusable in others. I don't we can make any conclusive verdict about the promise of ML for radiography right now; the life-critical nature of the application it's in the unusable middle, but it might get better in a few years or it might not. Time will tell.
Ekaros 9mo ago
I think finding that AI or at least specific model sold to be able to do something can't reasonable do it is entirely reasonable thing to publish. In the end it is on the model marketer to prove that what they sell can do what it says. And counter examples is fully valid thing to then release.

shevy-java 9mo ago

So basically AI kills people. This is Skynet 2.0 or 3.0. But shit. James Cameron may have to redo The Terminator, to include AI. Then again, who would watch such a movie?

NVHacker 9mo ago

Not all AIs are created equal.

nephihaha 9mo ago

As someone else has pointed out, I would like to know how this compares to humans. I hope something good comes out of this, as I have known women whose lives were deeply affected by this.

davidguetta 9mo ago

This is a terrible article. "One AI is not great" is not an interesting finding and certainly not conclusive of "AI can't help or do the job". It's like saying "some dude can't detect breast cancer" and suggesting all humans are useless.

zqy123007 9mo ago

okay guys, I developed AI mammo screening product, let me clear things up. you read it wrong, and I don't blame you. I doubt whoever wrote this actually have a good understanding of the numbers. the setup: 1. 400s confirmed patients 2. AI reads Mammography ONLY and missed 1/3 3. on those AI missed patients, radiologists do a second read on MRI, which is the gold standard for differential. evidence: the referenced paper at the bottom <Added value of diffusion-weighted imaging in detecting breast cancer missed by artificial intelligence-based mammography.> So, the whole point it (or its reference paper) is trying to make is: Mammography sucks, MRI is much better, which is a KNOWN FACT. Now, let me give you some more background missing from the paper: 1. Why does Mammography suck? well, go google/gpt some images, its essentially X-ray for the breast, which compress 3D volumes into 2D average poole plane, which is infomation lossy. SO, AI or not, the sensitivity is limited by the modality. 2. How bad/good is Mammography AI? I would say 80~85% sensitivity agaist very thorough+experienced radiologist without making unbearable amount of FP, that probably translates to 2/3 sensitivity against real cancer cohert, so the referenced number is about right. 3. Mammography sucks, what's the point? its cheap AND fast, you can probably do walk-in and get interpretation back in hours. Whereas MRI you probably need to schedule 2 weeks ahead if not MORE. For a yearly screening, it works for the majority of polulation. and final pro tip, 1. breast tumor is more pravelent than you think (maybe 1 in 2 by age 70+) 2. most guides recommend women with 45+ do yearly checkup 3. if you have dense breast (basically small and firm), add ultrasound screening to make sure. 4. breast feeding does good for both the mother and child, do that. peace & love

overtone1000 9mo ago
Radiologist here, thanks for posting this because I was biting my tongue. I'll supplement by directing others to consider how number needed to screen may be a more useful metric than mammographic sensitivity when making policy decisions. They're related, obviously, but only one of them concerns outcomes.

titaniumrain 9mo ago

The missed cases should be attributed to the specific model deployed in the product, not to AI as a general concept. Framing this limitation under a broad and alarming title is therefore misleading and unnecessary.

samrus 9mo ago

Whats the baseline? How many did a human being get? We need to compare this to baseline to know if its good or bad

directevolve 9mo ago
As there were no healthy controls or comparison with blinded radiologist performance, it is not possible to answer those questions using this study. The point was to evaluate AI sensitivity, which is useful. However, without healthy controls, it is not possible to determine specificity, which is a crucial statistic since most women do not have breast cancer.

mannyv 9mo ago

Obviously we need more breasts "for training."