14 ms·

Half of women will get a false positive 3D mammogram, study finds

256 points by samizdis 5y ago

haldujai 5y ago
This isn't quite as big of a problem as they're portraying and there is a lot of misinformation in these comments. For context I'm a radiologist at one of the academic centres involved in the ongoing TMIST (3D mammography) trial. For starters, false positives are an inherent part of medical diagnosis. Interpretation and diagnosis is probabilistic based on ROC curves balancing different sensitivities and specificities balanced with disease prevalence as well as the significance of missed/delayed diagnosis. Every concern that has been raised in these comments is factored into our reporting system. All radiologists use the BI-RADS reporting system which is internationally validated and expresses probabilities of malignancy to guide further management based on specific imaging. features that have been validated against pathologic data from excised surgical specimens and biopsies. This was done to homogenize reporting practices. For example a BI-RADS 5 lesion, which is based on specific imaging features not a gestalt/expert opinion, denotes a 95% probability of malignancy and would generally result urgent biopsy + surgical consultation. A BI-RADS 3 lesion has a 2% chance of malignancy and would generally be followed with serial imaging for a specified interval. BI-RADS 4 is the middle, this is broken down into A/B/C but essentially anything with 3%+ probability goes for biopsy. That means we expect up to 97% of the lesions we biopsy will have been false positives. This is intentional and desired, this is because the harms of missing breast cancer is horrible, early detection results in treatment with lumpectomy + radiation vs advanced breast cancer needing systemic therapies and radical resections. Second, what happens when there is an abnormality on a screening mammogram is not straight to biopsy. We use additional mammography views and ultrasound to help us find out what's going on before making that decision. We also have MRI to troubleshoot and increasingly for screening, certainly for high risk and extremely dense breasts. Tomosynthesis or 3D mammography is not a cure-all for false positives, it's also not intended for dense breasts. The point of Tomosynthesis is to reduce callbacks for overlapping breast tissue which can sometimes look like a real cancer, and to increase sensitivity for architectural distortion (an occult or infiltrative cancer distorting the fibroglandular architecture) which can be really hard to pick up on 2D. It works really great for that. Patients with extremely dense breasts should really be getting screening MRIs (US as a lesser alternative) in today's age and this is happening with increasing frequently. Further reading: https://en.wikipedia.org/wiki/Receiver_operating_characteristic https://en.wikipedia.org/wiki/Receiver_operating_characteris... https://www.acr.org/Clinical-Resources/Reporting-and-Data-Systems/Bi-Rads https://www.acr.org/Clinical-Resources/Reporting-and-Data-Sy... https://www.nejm.org/doi/full/10.1056/NEJMoa1903986 https://www.nejm.org/doi/full/10.1056/NEJMoa1903986 https://www.cancer.gov/about-cancer/treatment/clinical-trials/nci-supported/tmist https://www.cancer.gov/about-cancer/treatment/clinical-trial...
- dontreact 5y ago
  For starters, false positives are an inherent part of medical diagnosis. Interpretation and diagnosis is probabilistic based on ROC curves balancing different sensitivities and specificities balanced with disease prevalence as well as the significance of missed/delayed diagnosis. I totally get this and it's technical terminology. I think my takeaway from trying to explain why this isn't a big deal on this thread is that calling a tomo or mammo "positive" is just a recipe for confusion. I understand why it's done this way historically, but the idea of a mammo or tomo being "positive for cancer" makes no sense since there will almost always be either an ultrasound a biopsy or something else before a diagnosis is made. It's a test for whether a more invasive/expensive test is worth it, not a test for cancer. Curious what your opinion is on AI for DBT? I think there is great long term potential here, even more so than the potential that we have seen so far in mammo (such as https://www.nature.com/articles/s41586-019-1799-6 https://www.nature.com/articles/s41586-019-1799-6), because it's easier for AI to thoroughly look at things in 3d and spot new patterns that are not obvious to human eyes.
  - haldujai 5y ago
    AI in medical imaging has been very limited and ineffective thus far. My institution pays for an expensive lesion detection / mammography CAD software package but it generates so many false positives I don’t use it in my practice. Overall having to review an extra n pseudolesions per scan results in decision fatigue and increases my miss rate, at least in my experience. There are several issues with AI when it comes to radiology that I can’t address with a single comment but given your question my biggest criticism would be that we don’t have a sensitivity problem in medicine. What we have is largely a specificity problem, but that’s inherent with imaging which is not a ground truth representation of a physical entity but rather how this tissue attenuates photons (for mammo). AI, like self driving cars, has to be an order of magnitude “safer” which generally translates into decreased specificity. What I mean by this is that I don’t miss a significant number of breast cancers, in fact it’s very very low. So an AI to improve my detection is pointless to me, what I really want is to recall less patients but AI won’t necessarily help with that because it’s inherent limitations of the imaging modality (scar and cancer look the same). I think you are in the right track that AI looking at information humans are not currently looking at is the future / next step. Not to replace a task already handled by physicians who have the ability to integrate disparate health records (I.e. does this patient have easier access to breast MRI or biopsy given location/insurance/biopsy schedule/MRI schedule/hospital resources and what are her personal goals of care to help me decide what to do with this lesion) that is currently silo’s and inaccessible to AI.
asdfasgasdgasdg 5y ago
Some clarifications: "Half" here is the 10 year cumulative probability of at least one false positive if you get a screening each year. The status quo ten-year false positive rate with the test used before 3d mammograms (digital mammography) is 56.3, and the false positive rate with 3d mammograms (aka tomosynthesis) is 49.6. So the study is actually reporting an improvement vs the previous state of the art! It's weird that the article is written to convey that the false positive rate of the new technique is a drawback when it's actually a benefit. (This comment has been heavily edited thanks to a correction I received below.)
- barbegal 5y ago
  I think you've got the numbers mixed up according to the paper https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2790521?utm_source=For_The_Media&utm_medium=referral&utm_campaign=ftm_links&utm_term=032522 https://jamanetwork.com/journals/jamanetworkopen/fullarticle... "tomosynthesis [3D mammogram] vs digital mammography [previous test] for all outcomes: 49.6% vs 56.3%"
  - asdfasgasdgasdg 5y ago
    Yeah, I mixed up which name applied to the new technique. To be clear, I don't know the space, I just read the study and assumed that "digital mammography" sounded more 3d-ish than "tomosythesis". I'll update my comment accordingly. Edit: oh man. Now I see that I mixed up the direction too! What in the heck. The article was written like there was a big problem with 3d mammograms so I assumed it was performing worse on this metric than the baseline. But in fact it's better?
- rkangel 5y ago
  I think that the title is actually a pretty clear and accurate statement. [If you assume that all women have mammograms at the normal cadence, over a period of 10 years then] half of women will get a false positive.
  - asdfasgasdgasdg 5y ago
deleted 5y ago
[deleted]
Johnny555 5y ago
My wife got a false positive a few years back, her doctor called her personally to discuss and reassure her that it was not necessarily a problem, just an area of suspicion. She wasn't all that anxious or worried about it. The worst thing she was worried about was the chance of needing a biopsy, not the chance of actual breast cancer. She went in for a followup ultrasound scan and was cleared. Given the survivability of early stage breast cancer versus late stage, a high false positive rate doesn't seem worse than the alternative of missing early stage breast cancer. Obviously, better imaging or other tools would be even better.
- tomrod 5y ago
  This is really the tradeoff we face in the binary classification space. Do we want false positives or false negative reduced? There are more sophisticated approaches, but if the costs in one direction are magnitudes more than in another, it becomes a clear preference.
- orzig 5y ago
  There's also a subtle clinical communication aspect to this. I once had a doctor message me shortly after a routine test and say "I need you to come back tomorrow", and my pregnant wife was losing it until we could get through to him and find out that he just wanted to ensure I hadn't gotten an infection during the test. My wife wasn't wrong to be upset! Several hours of suffering could have been avoided with a good pamphlet.
  - colinmhayes 5y ago
    Yep, I've got family members who are doctors and they always tell me that bedside manner is the most important part of their job. They have to go to continual learning conferences every year and the biggest part of them is usually bedside manner stuff.
  - Johnny555 5y ago
    Yeah, communication is key - no doubt it would have been a different story if she got a voice-mail from her doctor's office saying "We found a suspicious lump in your mammogram, we're going to need you to come in for a followup tomorrow". She probably would have had a sleepless night wondering how serious it was.
cupofpython 5y ago
https://www.breastcancer.org/research-news/3d-mammos-outperform-2d-per-5-yr-study https://www.breastcancer.org/research-news/3d-mammos-outperf... https://www.cancer.org/cancer/breast-cancer/screening-tests-and-early-detection/mammograms/limitations-of-mammograms.html https://www.cancer.org/cancer/breast-cancer/screening-tests-... Related articles on the subject. Something that should be kept in mind is that false positives should always be talked about alongside false negatives. There is no perfect test, and often in medicine there is an increase in one of those in order to decrease the other. "A false-negative mammogram looks normal even though breast cancer is present. Overall, screening mammograms miss about 1 in 8 breast cancer" It isn't discussed in the OP article, and I could not easily find something that differentiates false-negative rates in 2D vs 3D mammograms, but the cost of over-diagnosis might be worth it if that means less cancers go unnoticed in screenings.
blackbear_ 5y ago
Note that this is not due to bad tests, just a counter-intuitive probabilistic result happening when one tries to detect an a priori very rare condition. It follows from a straightforward application of Bayes formula: - Suppose a person has 1% chance of getting cancer, so p(cancer) = 0.01 - Suppose that the test has 99% sensitivity and specificity, i.e., p(positive | cancer) = p(negative | no cancer) = 0.99 - To use Bayes we first need the probability of a positive result regardless of disease status, i.e. p(positive) = p(positive | cancer) * p(cancer) + p(positive | no cancer) * p(no cancer) = 0.99 * 0.01 + 0.01 * 0.99 = 0.0198 - Then by Bayes we have p(cancer | positive) = p(positive | cancer) * p(cancer) / p(positive) = 0.99 * 0.01 / 0.0198 = 0.5 And things get worse the rarer the condition is. When p(cancer) = 0.5 the computations above give p(cancer | positive) = 0.99, and when p(cancer) = 0.001 then p(cancer | positive) = 0.09. In other words, the rarer the condition, the more precise tests have to be.
- deleted 5y ago
  [deleted]
- hackernewds 5y ago
  So in summary, it is a bad test for a rare case? Am I missing anything?
  - ceras 5y ago
    I think there's one nuance to add: the test is based on your base rate expectation for how likely the given individual is to have the rare disease. The more evidence you have that this person is more likely to have the disease, the more useful the test becomes. Some examples: - if the disease is more common among people over age 65, it's more useful on people in that age group - if the person displays more symptoms associated with the disease, and not also associated with more common diseases, then it's more useful for someone with those symptoms - if a disease is common in an area of the world, and the person has traveled there, the test is more useful The more factors you have, the less likely a false positive. This is why it's often better to avoid medical tests unless there's a reason to suspect someone is at elevated risk for a disease: a positive result is more likely to be a false positive.
somenameforme 5y ago
There's a really interesting mathematical 'paradox' with false positives along the same line as the birthday paradox. It's an important one also that many doctors get wrong. Let's say there is a terrible disease that 0.01% (1 in 10,000) of the population is infected with, and we develop a test that is 99% accurate. You go to the doctor and get a test. Oh no, it's positive. What are the chances that it's a false positive? Intuition would tell you 1% since the test is 99% accurate. Intuition, as it often is, is wrong. Take a sample size of 'x', we'll use 10,000. How many people will be infected? 1. How many false positives will there be? 1% * 10,000 = 100. So there will be 100 false positives, and 1 real positive. So it turns out there's a greater than 99% chance that your result is a false positive, even though the test is genuinely 99% accurate. Here's a fun one. So we decide to run the test twice. And AGAIN it comes up positive. What are the odds that it's a false positive? Again, the exact same logic holds - there are 101 people we are testing. There will be about 1 false positive and about 1 real infection. So the odds of it being a false positive once again are about 50%.
- User23 5y ago
  A good example is screening men for breast cancer. It's possible for men to get breast cancer, but it's so vanishingly rare that the overwhelming majority of positive results will be false.
- reincarnate0x14 5y ago
  This is, incidentally, the same basic reason why so many knee-jerk policies produce systems that can never do anything but fail. When trying to find needles in hay stacks, since the needle (let's say terrorists trying to board airplanes) is so tiny relative to the hay (the millions of other people doing same), the people responsible for reacting are essentially trained only on false positives and still miss the real events when they (highly infrequently) occur. Obviously with low-impact false positives like requiring more expensive or invasive, but not really dangerous, additional medical tests this is an easily accepted and accounted for set of outcomes, but as we try to do things like apply ML to ever more specific cases of complex questions it's worth keeping in mind.
  - deleted 5y ago
giantg2 5y ago
I hate all these new 3D imaging things. I wonder how much difference there really is with the older tech based on looking at outcomes. For standard surveillance, does the new 3D dental xray actually benefit me? Granted the cost is low (<10% normal yearly exposure), but I still wonder.
- abxytg 5y ago
  Often it's a volumetric view of the same data, which is usually viewed as 2D slices. Doctors tend to prefer the slices.
  - giantg2 5y ago
    Yeah, but what are they doing with that to benefit me? Does it identify dental issues that traditional imaging misses? How common is that?
  - wincy 5y ago
    I remember almost fainting when my doctor showed me the MRI scans of my brain. It doesn’t bother me to see someone else’s scan, but I just couldn’t stomach looking at mine at all, seeing my optic nerve and my eyeballs and everything.
    - jhgb 5y ago
      And I got my MRI brain scan and I had absolutely zero emotional reaction to it, except for curiosity. But then again I'm a schizoid so that was very likely anyway...
- fedorareis 5y ago
  Well according to the article in this case 3D imaging is reducing false positives by about 6% which seems like an actual benefit. Would it be a benefit if it is easier to maintain or easier for the doctor to read?
- dontreact 5y ago
  It will take a long time to sort this out. I think that eventually, the ability for AI to review 3D in greater detail will improve on false positives and false negatives. For CT vs. XRay there have already been 2 long term randomized controlled trials that look at outcomes for lung cancer screening. PLCO, the XRay study, was not effective in improving long term outcomes like mortality. NLST, the CT study, showed a 20% reduction in mortality. With more modern techniques that actually look at the tumor in 3D NELSON in Europe has seen even better mortality reduction. DBT is newer so A) Radiologists are not as good at reading it yet B) the long term evidence hasn't accumulated yet, but I am definitely optimistic that it will present a big improvement over the previous state of the art. As mentioned elsewhere in the thread, this false positive number is already better than the existing state of the art for 2D.
abxytg 5y ago
Kind of a non sequitur but anyone else working on this stuff? I am writing a mammo rendering engine for the browser and it would be nice to have people to talk shop with. This shit is hard.
- husarcik 5y ago
  im a radiology resident interested in this space. Feel free to DM me on Twitter, @husarcik. :-)
tonymet 5y ago
i’m hoping with Covid more people are familiar with testing theory : ie sensitivity , specificity , false positives and false negatives these testing systems are just one of many indicators that are meant to be used . if used in isolation - very harmful outcomes will occur
- dontreact 5y ago
  Yes exactly. No one is going to put you on chemo because of a mammogram. There are several followup tests including ultrasound and biopsy.
  - tonymet 5y ago
    so the millions of false positives will mean hundreds who die from biopsy infection and hundreds given false chemo
- deleted 5y ago
  [deleted]
gzer0 5y ago
False-positive findings on screening mammography caused long-term psychosocial harm 3 years after a false-positive finding [1]. False positives also have a major economic impact, to the tune of $4 billion USD [2]. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3601385/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3601385/ [2] https://www.healthaffairs.org/doi/full/10.1377/hlthaff.2014.1087 https://www.healthaffairs.org/doi/full/10.1377/hlthaff.2014....
- dontreact 5y ago
  This is all certainly true but it's important to look at the bigger picture. What is the overall impact of catching cancer earlier and saving lives? That is enormous as well. One way this is sometimes talked about in public health literature is by looking at the QALYs of an intervention. On the whole, breast cancer screening does seem to be an effective intervention https://pubmed.ncbi.nlm.nih.gov/31098856/ https://pubmed.ncbi.nlm.nih.gov/31098856/
- stewbrew 5y ago
  That's why it is so important medical doctors comminicate this right. Unfortunately, most of them have a false understanding of the probability of cancer based on a positive (other than B5) mammography. But this is a solveable problem. The "4B$" is a rather useless number. The same people would probably argue it's much cheaper to go to your local shaman instead of a medical doctor.
dontreact 5y ago
It's good to be publishing stats like this so that the psychological harm of false positives is lower. It's scary to get a false positive. However, in my opinion, there has been a systemic resistance to screening because of statistics like this, and that is misguided. This is not the right number to use for determining whether or not screening is useful. A 3D mammogram will have false positives, yes. And some women will need to either get an uneccesary ultrasound (annoying, but not harmful) or worse, an uneccessary tissue sample (biopsy). This type of biopsy is done with a needle and is certainly uncomfortable, but carries little to no risk. This is all true. But it needs to be balanced against the fact that catching cancer early gives you an immensely better chance of surviving. The right numbers to look at are more like: How much does a screening program reduce overall mortality rates? How much does the screening program reduce breast cancer mortality rate? What are the impacts of this in terms of QALY (quality adjusted life years)? What are the impacts of the false positives in quality adjusted life years? I believe on balance the answer to these questions is that screening for breast cancer is on the whole beneficial. Here is one example study https://pubmed.ncbi.nlm.nih.gov/31098856/ https://pubmed.ncbi.nlm.nih.gov/31098856/ Now the question is, how will 3D affect these questions? That is as of yet unanswered as far as I know but we shouldn't let numbers like this influence the conversation prematurely.
- Aulig 5y ago
  You're totally right. However, the study you link does not discuss mortality while taking overdiagnosis into account from what I can tell. Other studies (first one I found: https://pubs.rsna.org/doi/full/10.1148/radiol.11110210 https://pubs.rsna.org/doi/full/10.1148/radiol.11110210 ) can't confirm a significant reduction in mortality.
  - dontreact 5y ago
    Yeah I just linked that study because it was looking at QALY which is an aggregate metric so it should in some sense take into account the effects of overdiagnosis even if overdiagnosis is not explicitly handled. Here is another study looking at all-cause mortality, which again does implicitly handle overdiagnosis. https://www.ajronline.org/doi/full/10.2214/AJR.14.12666 https://www.ajronline.org/doi/full/10.2214/AJR.14.12666
Aulig 5y ago
I looked into this recently. From what I can tell, no study could show a reduction in deaths from doing mammograms. This is because they detect many cancers that would go away on their own thus unnecessarily exposing you to chemotherapy which brings other health risks. A good starting point on this topic is this video: https://youtu.be/_sg14En-Z7A https://youtu.be/_sg14En-Z7A I was doubtful about the validity of that video at first, but everything he says is backed up by studies.
- dontreact 5y ago
  https://pubmed.ncbi.nlm.nih.gov/33141657/ https://pubmed.ncbi.nlm.nih.gov/33141657/ "Long-term follow-up of the UK Age RCT" "There was a statistically significant 25% reduction in mortality from breast cancers diagnosed during the intervention phase at 10 years' follow-up"
  - kmonad 5y ago
    Thank you.
  - stewbrew 5y ago
    This is specifically for an annual screening of women 40-49yrs. The reason why this article was written is because there is a scientific debate about whether the advantages outweigh the disadvantages in this age group. I don't think this article really settles this discussion. When you look at the CI you'll see it's rather big and the upper bound is only slightly below 1. But this is about screening of rather young women. Most established screening programs start at an age of 50. Still young. :-)
- stewbrew 5y ago
  Cancer usually doesn't go away just like that. There is a chance in situ tumours or benigne tumours "go away" and there is some discussion about what you should do with this type of tumour. Not every tumour is cancer. Anyway, even with cancer you can of course bet on dying of another cause before the cancer gets you and thus ignore medical therapy. Some people win this bet ... somehow.
SanchoPanda 5y ago
Here is a sample bayes rule calculcator made in my favorite web tool of all time, instacalc. This feels like the right way to do the math here. Random numbers provided, fill in your own as you deem appropriate. <https://instacalc.com/55494/embed?d=&c=QWN0dWFsX3Byb2JhYmlsaXR5ID0gMSV8UHJvYl90cnVlX3Bvc2l0aXZlID0gODAlfFByb2JfZmFsc2VfcG9zaXRpdmUgPSA5LjYlfC8vQ2hhbmNlIHBvc2l0aXZlIHRlc3QgbWVhbnMgcG9zaXRpdmUgcmVzdWx0fChyMSAqIHIyKS8oKHIxICogcjIpICsgcjMqKDEtcjEpKSAqIDEwMCAvLyAl&s=hhhhh&v=0.9 https://instacalc.com/55494/embed?d=&c=QWN0dWFsX3Byb2JhYmlsa...>
stewbrew 5y ago
Funny. The original article says "The findings of this study suggest that digital breast tomosynthesis is associated with a lower cumulative probability of false-positive results compared with digital mammography" Which is an important finding since today such screenings rely on digital mammography. So, tomography is good. For whatever reason, the cited article turns this upside down into: "Half of all women getting 3D mammograms will experience a false positive over a decade of annual screening" Which suggests that tomography performs really bad. If you do a mammography screening every two years, you do 5 mammographies in 10 years. With a false positive, you do 5 and an MRI. So what. Doctors should always tell their patients that a positive screening mammography doesn't necessarily imply they actually have cancer.
WaitWaitWha 5y ago
I am always (jadedly) suspicious with articles on 'false positives' within the medical field. Sure, statistics say that there are false positives. Personally, I rather have three false positives, than one false negative. Taken to a bit of an extreme, the outcome of first is money spent and stress, the outcome of the later is death. I am not a statistics to myself. I am sorry for the inconvenience to the medical field by having to re-test, and certainly feel for the insurance company to have to pay for a second, or third opinion. I care, I really do. Just, not that much. That said, it took me decades to recognize the need and add sympathy into my calculations.
matheusmoreira 5y ago
Of course. It's a high sensitivity test. It's meant to screen patients for more specific tests such as biopsies. You want it to be positive for anything that even remotely resembles breast cancer, the result can be confirmed by additional testing.
ggm 5y ago
Half of all women getting a 2d mammogram too. So.. the problem is not 3d vs 2d, from my reading of the axios story.
- stewbrew 5y ago
  Read the orig article. The pop-sci axios article misses the point.
xkbarkar 5y ago
Seems to me like the article is mostly pointing out the stress and out of pocket costs that come with all these false positives. An ultrasound is not intrusive at all, a biopsy however is. If I was a conspiracy theorist I'd say that a high amount of false positives are in fact an economic incentive for the medical community. And anyone having seen the hell metastatic breast cancer is, up close and personal, will insist that 1000 false positives is worth it to catch one case early on. Cancer awareness spam combined with obsessive screening and mild hostility towards women who do not frequently line up for ovarian/breast cancer testing (I don't see men being targeted nearly as much as women) is probably less about saving lives and more about making money imho. A biopsy is not at all harmless neither in the breast nor vagina. Do a quick internet search on prostate biopsy to see some gruesome numbers. All are pretty painful, prone to infection and a high chance of side effects. Some pretty severe, and they are not astronomically rare. The severe ones are in the 1% range. The fear of the brutality that is metastatic cancer is enough for few object to being routinely subjected and even coerced into this pretty dangerous and intrusive procedure. Those who do object are generally dismissed or even ridiculed. There is even a radiologist in the commentary here who sees no issue with the high rates of false positives as a potentially dangerous biopsy will quickly clear up the cancer scare... For vaginal scraping which is actually recommended as a screening measure for my age group 2-4 out of 100 women will have side effects. About 1 out of 100 will have severe side effects. Anecdotally. I have even had one doctor ( female if that matters ) tell me that after 50 it is probably best to have both my breasts and ovaries removed. As a preventative measure. This is the link I received from my doctor in Iceland about vaginal scraping and this is where I took my side effect statistics from. https://www.landspitali.is/library/Sameiginlegar-skrar/Gagnasafn/Sjuklingar-og-adstandendur/Sjuklingafraedsla---Upplysingarit/Kvennadeildir/utskaf_ur_legi_2021 https://www.landspitali.is/library/Sameiginlegar-skrar/Gagna... Medical doctors tend to terrify me. I really do avoid them unless I have something really concrete to show them that needs fixing and is not up for interpretation. EDIT: typos
- stewbrew 5y ago
  Every person has to decide for themselves whether he/she wants to do a screening examination. It's not for everyone and it is okay to say no. But I think you should try to find peace with medical doctors. Most of them just try to be helpful.

256 points by samizdis 5y ago

This isn't quite as big of a problem as they're portraying and there is a lot of misinformation in these comments. For context I'm a radiologist at one of the academic centres involved in the ongoing TMIST (3D mammography) trial. For starters, false positives are an inherent part of medical diagnosis. Interpretation and diagnosis is probabilistic based on ROC curves balancing different sensitivities and specificities balanced with disease prevalence as well as the significance of missed/delayed diagnosis. Every concern that has been raised in these comments is factored into our reporting system. All radiologists use the BI-RADS reporting system which is internationally validated and expresses probabilities of malignancy to guide further management based on specific imaging. features that have been validated against pathologic data from excised surgical specimens and biopsies. This was done to homogenize reporting practices. For example a BI-RADS 5 lesion, which is based on specific imaging features not a gestalt/expert opinion, denotes a 95% probability of malignancy and would generally result urgent biopsy + surgical consultation. A BI-RADS 3 lesion has a 2% chance of malignancy and would generally be followed with serial imaging for a specified interval. BI-RADS 4 is the middle, this is broken down into A/B/C but essentially anything with 3%+ probability goes for biopsy. That means we expect up to 97% of the lesions we biopsy will have been false positives. This is intentional and desired, this is because the harms of missing breast cancer is horrible, early detection results in treatment with lumpectomy + radiation vs advanced breast cancer needing systemic therapies and radical resections. Second, what happens when there is an abnormality on a screening mammogram is not straight to biopsy. We use additional mammography views and ultrasound to help us find out what's going on before making that decision. We also have MRI to troubleshoot and increasingly for screening, certainly for high risk and extremely dense breasts. Tomosynthesis or 3D mammography is not a cure-all for false positives, it's also not intended for dense breasts. The point of Tomosynthesis is to reduce callbacks for overlapping breast tissue which can sometimes look like a real cancer, and to increase sensitivity for architectural distortion (an occult or infiltrative cancer distorting the fibroglandular architecture) which can be really hard to pick up on 2D. It works really great for that. Patients with extremely dense breasts should really be getting screening MRIs (US as a lesser alternative) in today's age and this is happening with increasing frequently. Further reading: https://en.wikipedia.org/wiki/Receiver_operating_characteristic https://en.wikipedia.org/wiki/Receiver_operating_characteris... https://www.acr.org/Clinical-Resources/Reporting-and-Data-Systems/Bi-Rads https://www.acr.org/Clinical-Resources/Reporting-and-Data-Sy... https://www.nejm.org/doi/full/10.1056/NEJMoa1903986 https://www.nejm.org/doi/full/10.1056/NEJMoa1903986 https://www.cancer.gov/about-cancer/treatment/clinical-trials/nci-supported/tmist https://www.cancer.gov/about-cancer/treatment/clinical-trial...

dontreact 5y ago
For starters, false positives are an inherent part of medical diagnosis. Interpretation and diagnosis is probabilistic based on ROC curves balancing different sensitivities and specificities balanced with disease prevalence as well as the significance of missed/delayed diagnosis. I totally get this and it's technical terminology. I think my takeaway from trying to explain why this isn't a big deal on this thread is that calling a tomo or mammo "positive" is just a recipe for confusion. I understand why it's done this way historically, but the idea of a mammo or tomo being "positive for cancer" makes no sense since there will almost always be either an ultrasound a biopsy or something else before a diagnosis is made. It's a test for whether a more invasive/expensive test is worth it, not a test for cancer. Curious what your opinion is on AI for DBT? I think there is great long term potential here, even more so than the potential that we have seen so far in mammo (such as https://www.nature.com/articles/s41586-019-1799-6 https://www.nature.com/articles/s41586-019-1799-6), because it's easier for AI to thoroughly look at things in 3d and spot new patterns that are not obvious to human eyes.
- haldujai 5y ago
  AI in medical imaging has been very limited and ineffective thus far. My institution pays for an expensive lesion detection / mammography CAD software package but it generates so many false positives I don’t use it in my practice. Overall having to review an extra n pseudolesions per scan results in decision fatigue and increases my miss rate, at least in my experience. There are several issues with AI when it comes to radiology that I can’t address with a single comment but given your question my biggest criticism would be that we don’t have a sensitivity problem in medicine. What we have is largely a specificity problem, but that’s inherent with imaging which is not a ground truth representation of a physical entity but rather how this tissue attenuates photons (for mammo). AI, like self driving cars, has to be an order of magnitude “safer” which generally translates into decreased specificity. What I mean by this is that I don’t miss a significant number of breast cancers, in fact it’s very very low. So an AI to improve my detection is pointless to me, what I really want is to recall less patients but AI won’t necessarily help with that because it’s inherent limitations of the imaging modality (scar and cancer look the same). I think you are in the right track that AI looking at information humans are not currently looking at is the future / next step. Not to replace a task already handled by physicians who have the ability to integrate disparate health records (I.e. does this patient have easier access to breast MRI or biopsy given location/insurance/biopsy schedule/MRI schedule/hospital resources and what are her personal goals of care to help me decide what to do with this lesion) that is currently silo’s and inaccessible to AI.

asdfasgasdgasdg 5y ago

Some clarifications: "Half" here is the 10 year cumulative probability of at least one false positive if you get a screening each year. The status quo ten-year false positive rate with the test used before 3d mammograms (digital mammography) is 56.3, and the false positive rate with 3d mammograms (aka tomosynthesis) is 49.6. So the study is actually reporting an improvement vs the previous state of the art! It's weird that the article is written to convey that the false positive rate of the new technique is a drawback when it's actually a benefit. (This comment has been heavily edited thanks to a correction I received below.)

barbegal 5y ago
I think you've got the numbers mixed up according to the paper https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2790521?utm_source=For_The_Media&utm_medium=referral&utm_campaign=ftm_links&utm_term=032522 https://jamanetwork.com/journals/jamanetworkopen/fullarticle... "tomosynthesis [3D mammogram] vs digital mammography [previous test] for all outcomes: 49.6% vs 56.3%"
- asdfasgasdgasdg 5y ago
  Yeah, I mixed up which name applied to the new technique. To be clear, I don't know the space, I just read the study and assumed that "digital mammography" sounded more 3d-ish than "tomosythesis". I'll update my comment accordingly. Edit: oh man. Now I see that I mixed up the direction too! What in the heck. The article was written like there was a big problem with 3d mammograms so I assumed it was performing worse on this metric than the baseline. But in fact it's better?
rkangel 5y ago
I think that the title is actually a pretty clear and accurate statement. [If you assume that all women have mammograms at the normal cadence, over a period of 10 years then] half of women will get a false positive.
- asdfasgasdgasdg 5y ago

deleted 5y ago

[deleted]

Johnny555 5y ago

My wife got a false positive a few years back, her doctor called her personally to discuss and reassure her that it was not necessarily a problem, just an area of suspicion. She wasn't all that anxious or worried about it. The worst thing she was worried about was the chance of needing a biopsy, not the chance of actual breast cancer. She went in for a followup ultrasound scan and was cleared. Given the survivability of early stage breast cancer versus late stage, a high false positive rate doesn't seem worse than the alternative of missing early stage breast cancer. Obviously, better imaging or other tools would be even better.

tomrod 5y ago
This is really the tradeoff we face in the binary classification space. Do we want false positives or false negative reduced? There are more sophisticated approaches, but if the costs in one direction are magnitudes more than in another, it becomes a clear preference.
orzig 5y ago
There's also a subtle clinical communication aspect to this. I once had a doctor message me shortly after a routine test and say "I need you to come back tomorrow", and my pregnant wife was losing it until we could get through to him and find out that he just wanted to ensure I hadn't gotten an infection during the test. My wife wasn't wrong to be upset! Several hours of suffering could have been avoided with a good pamphlet.
- colinmhayes 5y ago
  Yep, I've got family members who are doctors and they always tell me that bedside manner is the most important part of their job. They have to go to continual learning conferences every year and the biggest part of them is usually bedside manner stuff.
- Johnny555 5y ago
  Yeah, communication is key - no doubt it would have been a different story if she got a voice-mail from her doctor's office saying "We found a suspicious lump in your mammogram, we're going to need you to come in for a followup tomorrow". She probably would have had a sleepless night wondering how serious it was.

cupofpython 5y ago

https://www.breastcancer.org/research-news/3d-mammos-outperform-2d-per-5-yr-study https://www.breastcancer.org/research-news/3d-mammos-outperf... https://www.cancer.org/cancer/breast-cancer/screening-tests-and-early-detection/mammograms/limitations-of-mammograms.html https://www.cancer.org/cancer/breast-cancer/screening-tests-... Related articles on the subject. Something that should be kept in mind is that false positives should always be talked about alongside false negatives. There is no perfect test, and often in medicine there is an increase in one of those in order to decrease the other. "A false-negative mammogram looks normal even though breast cancer is present. Overall, screening mammograms miss about 1 in 8 breast cancer" It isn't discussed in the OP article, and I could not easily find something that differentiates false-negative rates in 2D vs 3D mammograms, but the cost of over-diagnosis might be worth it if that means less cancers go unnoticed in screenings.

blackbear_ 5y ago

Note that this is not due to bad tests, just a counter-intuitive probabilistic result happening when one tries to detect an a priori very rare condition. It follows from a straightforward application of Bayes formula: - Suppose a person has 1% chance of getting cancer, so p(cancer) = 0.01 - Suppose that the test has 99% sensitivity and specificity, i.e., p(positive | cancer) = p(negative | no cancer) = 0.99 - To use Bayes we first need the probability of a positive result regardless of disease status, i.e. p(positive) = p(positive | cancer) * p(cancer) + p(positive | no cancer) * p(no cancer) = 0.99 * 0.01 + 0.01 * 0.99 = 0.0198 - Then by Bayes we have p(cancer | positive) = p(positive | cancer) * p(cancer) / p(positive) = 0.99 * 0.01 / 0.0198 = 0.5 And things get worse the rarer the condition is. When p(cancer) = 0.5 the computations above give p(cancer | positive) = 0.99, and when p(cancer) = 0.001 then p(cancer | positive) = 0.09. In other words, the rarer the condition, the more precise tests have to be.

deleted 5y ago
[deleted]
hackernewds 5y ago
So in summary, it is a bad test for a rare case? Am I missing anything?
- ceras 5y ago
  I think there's one nuance to add: the test is based on your base rate expectation for how likely the given individual is to have the rare disease. The more evidence you have that this person is more likely to have the disease, the more useful the test becomes. Some examples: - if the disease is more common among people over age 65, it's more useful on people in that age group - if the person displays more symptoms associated with the disease, and not also associated with more common diseases, then it's more useful for someone with those symptoms - if a disease is common in an area of the world, and the person has traveled there, the test is more useful The more factors you have, the less likely a false positive. This is why it's often better to avoid medical tests unless there's a reason to suspect someone is at elevated risk for a disease: a positive result is more likely to be a false positive.

somenameforme 5y ago

There's a really interesting mathematical 'paradox' with false positives along the same line as the birthday paradox. It's an important one also that many doctors get wrong. Let's say there is a terrible disease that 0.01% (1 in 10,000) of the population is infected with, and we develop a test that is 99% accurate. You go to the doctor and get a test. Oh no, it's positive. What are the chances that it's a false positive? Intuition would tell you 1% since the test is 99% accurate. Intuition, as it often is, is wrong. Take a sample size of 'x', we'll use 10,000. How many people will be infected? 1. How many false positives will there be? 1% * 10,000 = 100. So there will be 100 false positives, and 1 real positive. So it turns out there's a greater than 99% chance that your result is a false positive, even though the test is genuinely 99% accurate. Here's a fun one. So we decide to run the test twice. And AGAIN it comes up positive. What are the odds that it's a false positive? Again, the exact same logic holds - there are 101 people we are testing. There will be about 1 false positive and about 1 real infection. So the odds of it being a false positive once again are about 50%.

User23 5y ago
A good example is screening men for breast cancer. It's possible for men to get breast cancer, but it's so vanishingly rare that the overwhelming majority of positive results will be false.
reincarnate0x14 5y ago
This is, incidentally, the same basic reason why so many knee-jerk policies produce systems that can never do anything but fail. When trying to find needles in hay stacks, since the needle (let's say terrorists trying to board airplanes) is so tiny relative to the hay (the millions of other people doing same), the people responsible for reacting are essentially trained only on false positives and still miss the real events when they (highly infrequently) occur. Obviously with low-impact false positives like requiring more expensive or invasive, but not really dangerous, additional medical tests this is an easily accepted and accounted for set of outcomes, but as we try to do things like apply ML to ever more specific cases of complex questions it's worth keeping in mind.
- deleted 5y ago

giantg2 5y ago

I hate all these new 3D imaging things. I wonder how much difference there really is with the older tech based on looking at outcomes. For standard surveillance, does the new 3D dental xray actually benefit me? Granted the cost is low (<10% normal yearly exposure), but I still wonder.

abxytg 5y ago
Often it's a volumetric view of the same data, which is usually viewed as 2D slices. Doctors tend to prefer the slices.
- giantg2 5y ago
  Yeah, but what are they doing with that to benefit me? Does it identify dental issues that traditional imaging misses? How common is that?
- wincy 5y ago
  I remember almost fainting when my doctor showed me the MRI scans of my brain. It doesn’t bother me to see someone else’s scan, but I just couldn’t stomach looking at mine at all, seeing my optic nerve and my eyeballs and everything.
  - jhgb 5y ago
    And I got my MRI brain scan and I had absolutely zero emotional reaction to it, except for curiosity. But then again I'm a schizoid so that was very likely anyway...
fedorareis 5y ago
Well according to the article in this case 3D imaging is reducing false positives by about 6% which seems like an actual benefit. Would it be a benefit if it is easier to maintain or easier for the doctor to read?
dontreact 5y ago
It will take a long time to sort this out. I think that eventually, the ability for AI to review 3D in greater detail will improve on false positives and false negatives. For CT vs. XRay there have already been 2 long term randomized controlled trials that look at outcomes for lung cancer screening. PLCO, the XRay study, was not effective in improving long term outcomes like mortality. NLST, the CT study, showed a 20% reduction in mortality. With more modern techniques that actually look at the tumor in 3D NELSON in Europe has seen even better mortality reduction. DBT is newer so A) Radiologists are not as good at reading it yet B) the long term evidence hasn't accumulated yet, but I am definitely optimistic that it will present a big improvement over the previous state of the art. As mentioned elsewhere in the thread, this false positive number is already better than the existing state of the art for 2D.

abxytg 5y ago

Kind of a non sequitur but anyone else working on this stuff? I am writing a mammo rendering engine for the browser and it would be nice to have people to talk shop with. This shit is hard.

husarcik 5y ago
im a radiology resident interested in this space. Feel free to DM me on Twitter, @husarcik. :-)

tonymet 5y ago

i’m hoping with Covid more people are familiar with testing theory : ie sensitivity , specificity , false positives and false negatives these testing systems are just one of many indicators that are meant to be used . if used in isolation - very harmful outcomes will occur

dontreact 5y ago
Yes exactly. No one is going to put you on chemo because of a mammogram. There are several followup tests including ultrasound and biopsy.
- tonymet 5y ago
  so the millions of false positives will mean hundreds who die from biopsy infection and hundreds given false chemo
deleted 5y ago
[deleted]

gzer0 5y ago

False-positive findings on screening mammography caused long-term psychosocial harm 3 years after a false-positive finding [1]. False positives also have a major economic impact, to the tune of $4 billion USD [2]. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3601385/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3601385/ [2] https://www.healthaffairs.org/doi/full/10.1377/hlthaff.2014.1087 https://www.healthaffairs.org/doi/full/10.1377/hlthaff.2014....

dontreact 5y ago
This is all certainly true but it's important to look at the bigger picture. What is the overall impact of catching cancer earlier and saving lives? That is enormous as well. One way this is sometimes talked about in public health literature is by looking at the QALYs of an intervention. On the whole, breast cancer screening does seem to be an effective intervention https://pubmed.ncbi.nlm.nih.gov/31098856/ https://pubmed.ncbi.nlm.nih.gov/31098856/
stewbrew 5y ago
That's why it is so important medical doctors comminicate this right. Unfortunately, most of them have a false understanding of the probability of cancer based on a positive (other than B5) mammography. But this is a solveable problem. The "4B$" is a rather useless number. The same people would probably argue it's much cheaper to go to your local shaman instead of a medical doctor.

dontreact 5y ago

It's good to be publishing stats like this so that the psychological harm of false positives is lower. It's scary to get a false positive. However, in my opinion, there has been a systemic resistance to screening because of statistics like this, and that is misguided. This is not the right number to use for determining whether or not screening is useful. A 3D mammogram will have false positives, yes. And some women will need to either get an uneccesary ultrasound (annoying, but not harmful) or worse, an uneccessary tissue sample (biopsy). This type of biopsy is done with a needle and is certainly uncomfortable, but carries little to no risk. This is all true. But it needs to be balanced against the fact that catching cancer early gives you an immensely better chance of surviving. The right numbers to look at are more like: How much does a screening program reduce overall mortality rates? How much does the screening program reduce breast cancer mortality rate? What are the impacts of this in terms of QALY (quality adjusted life years)? What are the impacts of the false positives in quality adjusted life years? I believe on balance the answer to these questions is that screening for breast cancer is on the whole beneficial. Here is one example study https://pubmed.ncbi.nlm.nih.gov/31098856/ https://pubmed.ncbi.nlm.nih.gov/31098856/ Now the question is, how will 3D affect these questions? That is as of yet unanswered as far as I know but we shouldn't let numbers like this influence the conversation prematurely.

Aulig 5y ago
You're totally right. However, the study you link does not discuss mortality while taking overdiagnosis into account from what I can tell. Other studies (first one I found: https://pubs.rsna.org/doi/full/10.1148/radiol.11110210 https://pubs.rsna.org/doi/full/10.1148/radiol.11110210 ) can't confirm a significant reduction in mortality.
- dontreact 5y ago
  Yeah I just linked that study because it was looking at QALY which is an aggregate metric so it should in some sense take into account the effects of overdiagnosis even if overdiagnosis is not explicitly handled. Here is another study looking at all-cause mortality, which again does implicitly handle overdiagnosis. https://www.ajronline.org/doi/full/10.2214/AJR.14.12666 https://www.ajronline.org/doi/full/10.2214/AJR.14.12666

Aulig 5y ago

I looked into this recently. From what I can tell, no study could show a reduction in deaths from doing mammograms. This is because they detect many cancers that would go away on their own thus unnecessarily exposing you to chemotherapy which brings other health risks. A good starting point on this topic is this video: https://youtu.be/_sg14En-Z7A https://youtu.be/_sg14En-Z7A I was doubtful about the validity of that video at first, but everything he says is backed up by studies.

dontreact 5y ago
https://pubmed.ncbi.nlm.nih.gov/33141657/ https://pubmed.ncbi.nlm.nih.gov/33141657/ "Long-term follow-up of the UK Age RCT" "There was a statistically significant 25% reduction in mortality from breast cancers diagnosed during the intervention phase at 10 years' follow-up"
- kmonad 5y ago
  Thank you.
- stewbrew 5y ago
  This is specifically for an annual screening of women 40-49yrs. The reason why this article was written is because there is a scientific debate about whether the advantages outweigh the disadvantages in this age group. I don't think this article really settles this discussion. When you look at the CI you'll see it's rather big and the upper bound is only slightly below 1. But this is about screening of rather young women. Most established screening programs start at an age of 50. Still young. :-)
stewbrew 5y ago
Cancer usually doesn't go away just like that. There is a chance in situ tumours or benigne tumours "go away" and there is some discussion about what you should do with this type of tumour. Not every tumour is cancer. Anyway, even with cancer you can of course bet on dying of another cause before the cancer gets you and thus ignore medical therapy. Some people win this bet ... somehow.

SanchoPanda 5y ago

Here is a sample bayes rule calculcator made in my favorite web tool of all time, instacalc. This feels like the right way to do the math here. Random numbers provided, fill in your own as you deem appropriate. <https://instacalc.com/55494/embed?d=&c=QWN0dWFsX3Byb2JhYmlsaXR5ID0gMSV8UHJvYl90cnVlX3Bvc2l0aXZlID0gODAlfFByb2JfZmFsc2VfcG9zaXRpdmUgPSA5LjYlfC8vQ2hhbmNlIHBvc2l0aXZlIHRlc3QgbWVhbnMgcG9zaXRpdmUgcmVzdWx0fChyMSAqIHIyKS8oKHIxICogcjIpICsgcjMqKDEtcjEpKSAqIDEwMCAvLyAl&s=hhhhh&v=0.9 https://instacalc.com/55494/embed?d=&c=QWN0dWFsX3Byb2JhYmlsa...>

stewbrew 5y ago

Funny. The original article says "The findings of this study suggest that digital breast tomosynthesis is associated with a lower cumulative probability of false-positive results compared with digital mammography" Which is an important finding since today such screenings rely on digital mammography. So, tomography is good. For whatever reason, the cited article turns this upside down into: "Half of all women getting 3D mammograms will experience a false positive over a decade of annual screening" Which suggests that tomography performs really bad. If you do a mammography screening every two years, you do 5 mammographies in 10 years. With a false positive, you do 5 and an MRI. So what. Doctors should always tell their patients that a positive screening mammography doesn't necessarily imply they actually have cancer.

WaitWaitWha 5y ago

I am always (jadedly) suspicious with articles on 'false positives' within the medical field. Sure, statistics say that there are false positives. Personally, I rather have three false positives, than one false negative. Taken to a bit of an extreme, the outcome of first is money spent and stress, the outcome of the later is death. I am not a statistics to myself. I am sorry for the inconvenience to the medical field by having to re-test, and certainly feel for the insurance company to have to pay for a second, or third opinion. I care, I really do. Just, not that much. That said, it took me decades to recognize the need and add sympathy into my calculations.

matheusmoreira 5y ago

Of course. It's a high sensitivity test. It's meant to screen patients for more specific tests such as biopsies. You want it to be positive for anything that even remotely resembles breast cancer, the result can be confirmed by additional testing.

ggm 5y ago

Half of all women getting a 2d mammogram too. So.. the problem is not 3d vs 2d, from my reading of the axios story.

stewbrew 5y ago
Read the orig article. The pop-sci axios article misses the point.

xkbarkar 5y ago

Seems to me like the article is mostly pointing out the stress and out of pocket costs that come with all these false positives. An ultrasound is not intrusive at all, a biopsy however is. If I was a conspiracy theorist I'd say that a high amount of false positives are in fact an economic incentive for the medical community. And anyone having seen the hell metastatic breast cancer is, up close and personal, will insist that 1000 false positives is worth it to catch one case early on. Cancer awareness spam combined with obsessive screening and mild hostility towards women who do not frequently line up for ovarian/breast cancer testing (I don't see men being targeted nearly as much as women) is probably less about saving lives and more about making money imho. A biopsy is not at all harmless neither in the breast nor vagina. Do a quick internet search on prostate biopsy to see some gruesome numbers. All are pretty painful, prone to infection and a high chance of side effects. Some pretty severe, and they are not astronomically rare. The severe ones are in the 1% range. The fear of the brutality that is metastatic cancer is enough for few object to being routinely subjected and even coerced into this pretty dangerous and intrusive procedure. Those who do object are generally dismissed or even ridiculed. There is even a radiologist in the commentary here who sees no issue with the high rates of false positives as a potentially dangerous biopsy will quickly clear up the cancer scare... For vaginal scraping which is actually recommended as a screening measure for my age group 2-4 out of 100 women will have side effects. About 1 out of 100 will have severe side effects. Anecdotally. I have even had one doctor ( female if that matters ) tell me that after 50 it is probably best to have both my breasts and ovaries removed. As a preventative measure. This is the link I received from my doctor in Iceland about vaginal scraping and this is where I took my side effect statistics from. https://www.landspitali.is/library/Sameiginlegar-skrar/Gagnasafn/Sjuklingar-og-adstandendur/Sjuklingafraedsla---Upplysingarit/Kvennadeildir/utskaf_ur_legi_2021 https://www.landspitali.is/library/Sameiginlegar-skrar/Gagna... Medical doctors tend to terrify me. I really do avoid them unless I have something really concrete to show them that needs fixing and is not up for interpretation. EDIT: typos

stewbrew 5y ago
Every person has to decide for themselves whether he/she wants to do a screening examination. It's not for everyone and it is okay to say no. But I think you should try to find peace with medical doctors. Most of them just try to be helpful.