3 ms·

The point of this study is that it suggests that fully AI-automated mammography can currently deliver 70% sensitivity in detecting breast cancer using this mode

The point of this study is that it suggests that fully AI-automated mammography can currently deliver 70% sensitivity in detecting breast cancer using this model. It does not enable us to compare AI to unaided human performance. As this study did not include healthy controls, there is no false positive rate. The false positive rate is a crucial missing metric, since the vast majority of women do not have breast cancer. In nearly half the false negatives from both the mammogram and DWI datasets, the cancer was categorized as occult by two breast radiologists, meaning the cancer was invisible to a trained eye. The AI model's non-occult false negative rate on the mammography data is 19.3%. For that 19.3% figure, see Table 2: 68 non-occult in AI-missed cancer, 285 non-occult in AI-detected cancer. This study did not compare the AI to a radiologist on a mixed set of healthy and cancer images.

_heimdall 9mo ago
Its interesting to see this valid argument raised against this use of AI to identify breast cancer. The lack of control groups is one of the more common concerns raised related to vaccines as well, the argument lands like a lead balloon there.
- dgacmu 9mo ago
  Because it's generally unethical to not give someone a treatment known already to be safe and effective. Studies of new vaccines where there is not an existing vaccine _do_ use placebo controls. Heck, my son got placebo during moderna's pediatric covid vaccine trial (to our frustration. grin.) Subsequent trials generally compare against the best known current treatment as the control instead. This study has no such concerns. It's ethical to include images of non-cancerous breast tissue. The things are not comparable.
  - onetokeoverthe 9mo ago
    [dead]
  - _heimdall 9mo ago
    The covid vaccines were a whole different beast, though interesting case studies they were done under emergency authorization and didn't follow standard protocols. Vaccine studies today almost always use a previously approved vaccine as the "control" group. That isn't a true control and if you walk back the chain of approvals you'd be hard pressed to find a starting point that did use proper control groups. Anyway, my point here wasn't to directly debate vaccines themselves, only to point out that its interest to me as someone without a career in health to see the same effective argument used in two different scenarios with drastically different common responses.
    - dgacmu 9mo ago
      Right, but the people making the argument about vaccines don't understand the principles, because they're actually the same! 1) a double blind RCT with a placebo control is a very good way to understand the effectiveness of a treatment. 2) it's not always ethical to do that, because if you have an effective treatment, you must use it. Even without a placebo control you can still estimate both FN and FPs through careful study design, it's just harder and has more potential sources of error. A retrospective study is the usual approach. Here, the problem is they only included true positives in the retrospective study, so they missed the opportunity to measure false positives. And the problem with -that- is that it's very easy to have zero false negatives if you always say " it's positive". Almost every diagnostic instrument has something we call a receiver operating curve that trades off false positives for false negatives by changing the sensitivity for where you decide something is a positive. By omitting the false negatives, they present a very incomplete picture of the diagnostic capabilities. (In medicine you will often see the terms "sensitivity" and "selectivity" for how many TPs you detect and how many TNs you call negative. It's all part of the same type of characterization.)
      - _heimdall 9mo ago
        The two points you raise with regards to why vaccine or similar studies may be treat special, it doesn't replace the loss of data when a double blind study with a control or make estimates based on modelling indicate anything more than correlation. We may broadly agree that submitting a control group to a placebo treatment for a particular disease is immoral, but that doesn't mean such a study isn't necessary to prove out the efficacy or safety of the treatment. As for modelling, for example trying to estimate FN and FP, it can only ever indicate correlation at best and will never indicate likely causation.
- benterix 9mo ago
  > Its interesting to see this valid argument raised against this use of AI to identify breast cancer. The lack of control groups is one of the more common concerns raised related to vaccines as well, the argument lands like a lead balloon there. Not just vaccines, in each study on the effectiveness of a drug, especially when dealing with potentially life-threatening conditions, the same question is posed. From[0]: . . . ethical guidance permit the use of placebo controls in randomized trials when scientifically indicated in four cases: (1) when there is no proven effective treatment for the condition under study; (2) when withholding treatment poses negligible risks to participants; (3) when there are compelling methodological reasons for using placebo, and withholding treatment does not pose a risk of serious harm to participants; and, more controversially, (4) when there are compelling methodological reasons for using placebo, and the research is intended to develop interventions that can be implemented in the population from which trial participants are drawn, and the trial does not require participants to forgo treatment they would otherwise receive. [0] https://pmc.ncbi.nlm.nih.gov/articles/PMC3844122/ https://pmc.ncbi.nlm.nih.gov/articles/PMC3844122/
- directevolve 9mo ago
  Vaccine studies use a different experimental design known as “longitudinal,” meaning they follow people over time. This study did not do that. It’s still a valid design, just limited in what it tells us.
nabla9 9mo ago
As it was retrospective study, I really hope they made sure that test images were not in a training set of the algorithm. If they were, the whole study is meaningless.
gcr 9mo ago
What sensitivity / specificity are trained radiologists able to receive?
- kelseyfrog 9mo ago
  Great question. It prompted me to search for and find a history of sensitivity in mammography[1]. Their conclusion is that 39% is supported by evidence. Furthermore, there is a persistent erroneous belief that mammographic sensitivity is 90-95%. 1. https://pmc.ncbi.nlm.nih.gov/articles/PMC6640096/#R4 https://pmc.ncbi.nlm.nih.gov/articles/PMC6640096/#R4
vessenes 9mo ago
I'm not a medical researcher, but I am a computer guy; I was struck by something very different in the papers - the abstracts at least refer to "AI CAD" as what they're testing - no software information, no versioning - on the CS side, this stuff is of paramount importance to make sure we know how the software performs. On the medical side, we need statistically significant tests that physicians can know and rely on - this paper was likely obsolete when it was published, depending on what "AI CAD" means in practice. I think this impedance mismatch between disciplines is pretty interesting; any thoughts from someone who understands the med side better?
- mattkrause 9mo ago
  The link is essentially a press release. The information you want is (sorta) in the actual paper it describes *. "The images were analyzed using a commercially available AI-CAD system (Lunit INSIGHT MMG, version 1.1.7.0; Lunit Inc.), developed with deep convolutional neural networks and validated in multinational studies [1, 4]." It's presumably a proprietary model, so you're not going to get a lot more information about it, but it's also one that's currently deployed in clinics, so...it's arguably a better comparison than a SOTA model some lab dumped on GitHub. I'd add that the post headline is also missing the point of the article: many of the missed cases can be detected with a different form of imaging. It's not really meant to be a model shoot-out style paper. * Kim, J. Y., Kim, J. J., Lee, H. J., Hwangbo, L., Song, Y. S., Lee, J. W., Lee, N. K., Hong, S. B., & Kim, S. (2025). Added value of diffusion-weighted imaging in detecting breast cancer missed by artificial intelligence-based mammography. La Radiologia medica, 10.1007/s11547-025-02161-1. Advance online publication. https://doi.org/10.1007/s11547-025-02161 https://doi.org/10.1007/s11547-025-02161

_heimdall 9mo ago

Its interesting to see this valid argument raised against this use of AI to identify breast cancer. The lack of control groups is one of the more common concerns raised related to vaccines as well, the argument lands like a lead balloon there.

dgacmu 9mo ago
Because it's generally unethical to not give someone a treatment known already to be safe and effective. Studies of new vaccines where there is not an existing vaccine _do_ use placebo controls. Heck, my son got placebo during moderna's pediatric covid vaccine trial (to our frustration. grin.) Subsequent trials generally compare against the best known current treatment as the control instead. This study has no such concerns. It's ethical to include images of non-cancerous breast tissue. The things are not comparable.
- onetokeoverthe 9mo ago
  [dead]
- _heimdall 9mo ago
  The covid vaccines were a whole different beast, though interesting case studies they were done under emergency authorization and didn't follow standard protocols. Vaccine studies today almost always use a previously approved vaccine as the "control" group. That isn't a true control and if you walk back the chain of approvals you'd be hard pressed to find a starting point that did use proper control groups. Anyway, my point here wasn't to directly debate vaccines themselves, only to point out that its interest to me as someone without a career in health to see the same effective argument used in two different scenarios with drastically different common responses.
  - dgacmu 9mo ago
    Right, but the people making the argument about vaccines don't understand the principles, because they're actually the same! 1) a double blind RCT with a placebo control is a very good way to understand the effectiveness of a treatment. 2) it's not always ethical to do that, because if you have an effective treatment, you must use it. Even without a placebo control you can still estimate both FN and FPs through careful study design, it's just harder and has more potential sources of error. A retrospective study is the usual approach. Here, the problem is they only included true positives in the retrospective study, so they missed the opportunity to measure false positives. And the problem with -that- is that it's very easy to have zero false negatives if you always say " it's positive". Almost every diagnostic instrument has something we call a receiver operating curve that trades off false positives for false negatives by changing the sensitivity for where you decide something is a positive. By omitting the false negatives, they present a very incomplete picture of the diagnostic capabilities. (In medicine you will often see the terms "sensitivity" and "selectivity" for how many TPs you detect and how many TNs you call negative. It's all part of the same type of characterization.)
    - _heimdall 9mo ago
      The two points you raise with regards to why vaccine or similar studies may be treat special, it doesn't replace the loss of data when a double blind study with a control or make estimates based on modelling indicate anything more than correlation. We may broadly agree that submitting a control group to a placebo treatment for a particular disease is immoral, but that doesn't mean such a study isn't necessary to prove out the efficacy or safety of the treatment. As for modelling, for example trying to estimate FN and FP, it can only ever indicate correlation at best and will never indicate likely causation.
benterix 9mo ago
> Its interesting to see this valid argument raised against this use of AI to identify breast cancer. The lack of control groups is one of the more common concerns raised related to vaccines as well, the argument lands like a lead balloon there. Not just vaccines, in each study on the effectiveness of a drug, especially when dealing with potentially life-threatening conditions, the same question is posed. From[0]: . . . ethical guidance permit the use of placebo controls in randomized trials when scientifically indicated in four cases: (1) when there is no proven effective treatment for the condition under study; (2) when withholding treatment poses negligible risks to participants; (3) when there are compelling methodological reasons for using placebo, and withholding treatment does not pose a risk of serious harm to participants; and, more controversially, (4) when there are compelling methodological reasons for using placebo, and the research is intended to develop interventions that can be implemented in the population from which trial participants are drawn, and the trial does not require participants to forgo treatment they would otherwise receive. [0] https://pmc.ncbi.nlm.nih.gov/articles/PMC3844122/ https://pmc.ncbi.nlm.nih.gov/articles/PMC3844122/
directevolve 9mo ago
Vaccine studies use a different experimental design known as “longitudinal,” meaning they follow people over time. This study did not do that. It’s still a valid design, just limited in what it tells us.

nabla9 9mo ago

As it was retrospective study, I really hope they made sure that test images were not in a training set of the algorithm. If they were, the whole study is meaningless.

gcr 9mo ago

What sensitivity / specificity are trained radiologists able to receive?

kelseyfrog 9mo ago
Great question. It prompted me to search for and find a history of sensitivity in mammography[1]. Their conclusion is that 39% is supported by evidence. Furthermore, there is a persistent erroneous belief that mammographic sensitivity is 90-95%. 1. https://pmc.ncbi.nlm.nih.gov/articles/PMC6640096/#R4 https://pmc.ncbi.nlm.nih.gov/articles/PMC6640096/#R4

vessenes 9mo ago

I'm not a medical researcher, but I am a computer guy; I was struck by something very different in the papers - the abstracts at least refer to "AI CAD" as what they're testing - no software information, no versioning - on the CS side, this stuff is of paramount importance to make sure we know how the software performs. On the medical side, we need statistically significant tests that physicians can know and rely on - this paper was likely obsolete when it was published, depending on what "AI CAD" means in practice. I think this impedance mismatch between disciplines is pretty interesting; any thoughts from someone who understands the med side better?

mattkrause 9mo ago
The link is essentially a press release. The information you want is (sorta) in the actual paper it describes *. "The images were analyzed using a commercially available AI-CAD system (Lunit INSIGHT MMG, version 1.1.7.0; Lunit Inc.), developed with deep convolutional neural networks and validated in multinational studies [1, 4]." It's presumably a proprietary model, so you're not going to get a lot more information about it, but it's also one that's currently deployed in clinics, so...it's arguably a better comparison than a SOTA model some lab dumped on GitHub. I'd add that the post headline is also missing the point of the article: many of the missed cases can be detected with a different form of imaging. It's not really meant to be a model shoot-out style paper. * Kim, J. Y., Kim, J. J., Lee, H. J., Hwangbo, L., Song, Y. S., Lee, J. W., Lee, N. K., Hong, S. B., & Kim, S. (2025). Added value of diffusion-weighted imaging in detecting breast cancer missed by artificial intelligence-based mammography. La Radiologia medica, 10.1007/s11547-025-02161-1. Advance online publication. https://doi.org/10.1007/s11547-025-02161 https://doi.org/10.1007/s11547-025-02161