4 ms·

You have to be careful with screening tests. Say that this test has a false positive 1 in 1000 times. If you test 100,000 people, you'll get 100 positives tha

by mlyle 2y ago

You have to be careful with screening tests. Say that this test has a false positive 1 in 1000 times. If you test 100,000 people, you'll get 100 positives that need invasive further testing and followup, and 5 real pancreatic cancer cases. Society will pay for 100,000 tests, and 105 cases of followup. You may cause lasting harm to some of those 105 people. And then it's not clear if you can improve the survival of the 5 pancreatic cancer cases much. They'll live longer after diagnosis (because you diagnosed earlier) but not necessarily longer overall. (One other screening effect: You'll find more "real cancer" that is so slow growing that it may have always remained subclinical before the more sensitive testing; And the most serious cancers, you won't find so much sooner, because they grow so much in the interval between tests.)

GeekyBear 2y ago
You would need to take into account how aggressive a given cancer is and our ability to treat it. For instance, prostate cancer blood screening often led to radical treatments that are no longer thought to be worth it for most people. > most prostate cancer grows so slowly, if it grows at all, that other illnesses are likely to prove lethal first https://www.nytimes.com/2023/05/08/health/prostate-cancer-screening.html https://www.nytimes.com/2023/05/08/health/prostate-cancer-sc... In the case of pancreatic cancer, it is much more aggressive and you need to catch it early.
- mlyle 2y ago
  > In the case of pancreatic cancer, it is much more aggressive and you need to catch it early. It's not clear that the cancers that you would find early with a more sensitive test are those more aggressive cancers. The pancreatic cancers we find with our current detection (generally after becoming symptomatic) are typically quite aggressive. But are they all the cancers? Likewise, if the cancer is aggressive, it can grow quite a bit between screening intervals and not be found all that early. (Part of why we think that "finding cancer early" is such a benefit is that because the smaller/earlier cancers we find are less aggressive than the cancers that we first find when they're huge and spread. There is definitely an effect from earlier detection but our estimate of it has been confused by this effect.) As we've increased cancer screening, we've found that survival rates have gone up, as have survival times after detection... but unfortunately we've often also found that the screening doesn't always reduce the number of people dying of that cancer at a certain age. Instead, you find more cancers, and you find them earlier so more people live to 5 years, even if you've changed nothing. Cancer treatment has gotten better, but most of the benefits we have expected from better cancer screening have not materialized. Finding pancreatic cancer early sounds good. And it may be able to reduce mortality from pancreatic cancer, but it's not a sure thing.
- slashdev 2y ago
  My grandfather (a doctor) always used to say this. There’s also an aggressive fast growing kind of prostate cancer, but treatment basically does nothing for survival rates (or at least that was the case decades ago when he was practicing.) So his advice was, don’t look, don’t treat. Either you have the slow one and treatment is harmful, or you have the fast one and you’re going to die soon anyway.
  - GeekyBear 2y ago
    Your grandfather's take has become increasingly accepted for prostate cancer. There is more of a watch and see attitude to make sure that the patient doesn't have a rare case of aggressive growth. As you mentioned, the outcomes aren't significantly different, regardless of how you treat it. From the article linked above: > Researchers followed more than 1,600 men with localized prostate cancer who, from 1999 to 2009, received what they called active monitoring, a prostatectomy or radiation with hormone therapy. Over an exceptionally long follow-up averaging 15 years, fewer than 3 percent of the men, whose average age at diagnosis was 62, had died of prostate cancer. The differences between the three treatment groups were not statistically significant.
    - slashdev 2y ago
      The irony is he died of prostate cancer. He ignored his own advice and treated it. It did not change the outcome or buy him much time, if any.
- hsuduebc2 2y ago
  You can just run test multiple times to eradicate this possibility or you can confirm it with another method.
  - mlyle 2y ago
    > You can just run test multiple times to eradicate this possibility The measurements are not independent and the quality of the measurement is not improved by this. > you can confirm it with another method. Yes. And usually the other method is invasive and expensive and bears some risk. And then you get results like the blood test saying "very likely cancer" and the biopsy saying "uh, probably not?" that you need to decide what to do with.
    - hsuduebc2 2y ago
      I'm confused. If this blood test gave a false positive that wasn't due to an anomaly in the blood itself, then why can't we assume that the likelihood of getting a false positive twice is lower than getting it once?
      - mlyle 2y ago
        > I'm confused. If this blood test gave a false positive that wasn't due to an anomaly in the blood itself, That's the errant part. A small amount of the false positives will be because of lab issues. The rest will be because this patient is different in some way, but not all of them are cancer. Medicine doesn't have very many perfectly specific tests. So you have a patient who has some weird enzymes around because they're genetically different, for example, and they always pop positive on this particular test. Or has an unusual diet that causes some other non-tested-for-enzyme level to be high enough to set off this sensor. Or whatever. In this case, the specificity is 98%, so this false positive rate is about 2%.
caesil 2y ago
Then let's take those things into account when calculating what tests to do. Surely, though, we can do better as a society than solving this with "no preemptive testing except for extreme risks".
- lm28469 2y ago
  > Surely, though, we can do better as a society We haven't even solved the most basic shit like shelter, food, education, &c for millions of people in the west, as a society we're faaaaaaaaar from universal yearly full health checkups. As an individual feel free to get private checks, they'll gladly take your money
  - aurizon 2y ago
    The fact that there are huge costs in the USA to even periodic medical checkups has severely impacted longevity in the USA to the point it ranks close to Cuba in longevity. Those with a health plan are close to the highest ranked nations. The poor without a plan at all are around ~4-5 less long lived. There is a nice rabbit hole in this data. https://www.google.com/search?q=longevity+charts&rlz=1C1CHZN_enCA1101CA1101&oq=longevity+charts&gs_lcrp=EgZjaHJvbWUyBggAEEUYOdIBCjE1MjMxajBqMTWoAgiwAgE&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=longevity+charts&rlz=1C1CHZN... This has a huge GDP cost in the USA, that needs to be addressed. The causes are big pharma/hospo/AMA/insuro/lobbyo.... One wonders why the AMA is there? - they limit the numbers of doctors trained in Universities/training hospitals to forestall price competition among doctors by various means. Dentists do the same.
    - toast0 2y ago
      Cost of a yearly checkup should be "taken care of", because Obamacare mandated free annual checkups, as long as you don't accidentally trigger any other billing codes while you're there. But, regardless of cost, there's a shortage of providers, so it's hard to schedule the checkup. And there's still a lot of uninsured people out there.
    - nradov 2y ago
Redoubts 2y ago
> You may cause lasting harm to some of those 105 people. Could you elaborate on this?
- jtc331 2y ago
  For example you might do surgery on people who wouldn’t benefit.
- daedrdev 2y ago
  you will do surgery on some of the 105 people. Some of them might die from complications, infections, etc or at least have lasting damage. Since several of the 5 people will not be any better off with treatment it's entirely possible that the screening produces palpably worse outcome. The earlier you screen, the worse this is, since the ratio of false positives vs true positives gets higher and higher, for example 1000 vs 5 or 10000 vs 5.
  - tptacek 2y ago
    It's also psychologically harmful to have the positive test hanging over your head. I'm nearing the age where doctors start harassing about colonoscopies. You can do an at-home test instead of the full procedure, and it has a very good chance of ruling out the need for a colonoscopy. But it also has a high false positive rate; there's a decent chance that you'll end up in a state of "need a colonoscopy, also a colon cancer screener flagged you". I'm dreading the colonoscopy prep, but I'm not doing the at-home thing.
    - jeswin 2y ago
      I've had colonoscopies twice. Was just half a day of inconvenience if you schedule it for the morning.
      - tptacek 2y ago
        The scope, right? Not the at-home test? The only thing about the scope that bugs me is the prep.
AustinDev 2y ago
Can't you just run the test again instead of doing a full follow up? 1/1000 * 1/1000 = 1/1,000,000
- okaram 2y ago
  You only get that probability if the test results are completely uncorrelated, chances are, they're not. I'd assume the chances of getting a second false positive if you already got one are much higher.
tptacek 2y ago
For people surprised by this argument, the phenomenon he's invoking here has a name: the Bayesian Base Rate Fallacy.
fragmede 2y ago
> Society will pay for 100,000 tests For better or worse, under the American healthcare system, the patient pays for those tests, sometimes covered by insurance. If the tests are paid for out of pocket by the patient, is there still such issue?
- s1artibartfast 2y ago
  I think the issue is exactly the same no matter who pays. To reframe it from the individual patient's perspective, when you take a test simply for the sake of screening, there is the chance you'll learn something true that helps you, and the chance you will learn something false that hurts you
- mlyle 2y ago
  The economic argument doesn't change whether it's a private cost or purely a social cost (private costs are included in social costs, since private expenditures are part of society's expenditures).
fallingknife 2y ago
You are assuming that those false positive rates are fixed, but they aren't. The "positive" criteria are done by an analysis exactly as sophisticated as a human scanning a list of numbers. The process is a joke and it needs to be improved by more data and better analysis, not this nonsensical "don't test people because they might be positive" argument.
- mlyle 2y ago
  No, I'm assuming there's a tradeoff between sensitivity (spotting cancers) and specificity (having your positive results actually be cancer). ANOVA to pick variables and then reasonably selecting thresholds is a fine process that avoids overfit. The big problem is, biology is messy and measuring lots of people to find correct thresholds is really expensive and time consuming. It's not really a technological problem, though technology has helped a little.
hammock 2y ago
How come nobody seems uses this kind of math when it came to COVID prophylactics? Or did they?
dillydogg 2y ago
There was a seminar given to the breast cancer society by an epidemiologist years who who presented them with a scenario: Prevalence of breast cancer: 1% Sensitivity (percent of people who have the disease test positive): 90% Specificity (percent of people without the disease test negative): 91% And asked, "How many people who test positive have the disease?" (i.e., positive predictive value) It's only 1/11. I think only 20% got it right (in a 4 answer multiple choice question)
- mlyle 2y ago
  Yah, I'm well familiar with the base rate fallacy and I still catch myself screwing it up. It's so unintuitive. I use mental math a lot to counter my intuition. If sensitivity is high and the base rate is low, you can approximate it with .01 / (1-.91). Or, mental math assuming 10,000 people is not unreasonable for your case (100 with disease, 90 true positives; 9900 * 9/100 false positives.. divide everything by 9 to make it easier, 10 true positives and 99 false positives, or 10/109 or say "9%".

by mlyle 2y ago

GeekyBear 2y ago

You would need to take into account how aggressive a given cancer is and our ability to treat it. For instance, prostate cancer blood screening often led to radical treatments that are no longer thought to be worth it for most people. > most prostate cancer grows so slowly, if it grows at all, that other illnesses are likely to prove lethal first https://www.nytimes.com/2023/05/08/health/prostate-cancer-screening.html https://www.nytimes.com/2023/05/08/health/prostate-cancer-sc... In the case of pancreatic cancer, it is much more aggressive and you need to catch it early.

mlyle 2y ago
> In the case of pancreatic cancer, it is much more aggressive and you need to catch it early. It's not clear that the cancers that you would find early with a more sensitive test are those more aggressive cancers. The pancreatic cancers we find with our current detection (generally after becoming symptomatic) are typically quite aggressive. But are they all the cancers? Likewise, if the cancer is aggressive, it can grow quite a bit between screening intervals and not be found all that early. (Part of why we think that "finding cancer early" is such a benefit is that because the smaller/earlier cancers we find are less aggressive than the cancers that we first find when they're huge and spread. There is definitely an effect from earlier detection but our estimate of it has been confused by this effect.) As we've increased cancer screening, we've found that survival rates have gone up, as have survival times after detection... but unfortunately we've often also found that the screening doesn't always reduce the number of people dying of that cancer at a certain age. Instead, you find more cancers, and you find them earlier so more people live to 5 years, even if you've changed nothing. Cancer treatment has gotten better, but most of the benefits we have expected from better cancer screening have not materialized. Finding pancreatic cancer early sounds good. And it may be able to reduce mortality from pancreatic cancer, but it's not a sure thing.
slashdev 2y ago
My grandfather (a doctor) always used to say this. There’s also an aggressive fast growing kind of prostate cancer, but treatment basically does nothing for survival rates (or at least that was the case decades ago when he was practicing.) So his advice was, don’t look, don’t treat. Either you have the slow one and treatment is harmful, or you have the fast one and you’re going to die soon anyway.
- GeekyBear 2y ago
  Your grandfather's take has become increasingly accepted for prostate cancer. There is more of a watch and see attitude to make sure that the patient doesn't have a rare case of aggressive growth. As you mentioned, the outcomes aren't significantly different, regardless of how you treat it. From the article linked above: > Researchers followed more than 1,600 men with localized prostate cancer who, from 1999 to 2009, received what they called active monitoring, a prostatectomy or radiation with hormone therapy. Over an exceptionally long follow-up averaging 15 years, fewer than 3 percent of the men, whose average age at diagnosis was 62, had died of prostate cancer. The differences between the three treatment groups were not statistically significant.
  - slashdev 2y ago
    The irony is he died of prostate cancer. He ignored his own advice and treated it. It did not change the outcome or buy him much time, if any.
hsuduebc2 2y ago
You can just run test multiple times to eradicate this possibility or you can confirm it with another method.
- mlyle 2y ago
  > You can just run test multiple times to eradicate this possibility The measurements are not independent and the quality of the measurement is not improved by this. > you can confirm it with another method. Yes. And usually the other method is invasive and expensive and bears some risk. And then you get results like the blood test saying "very likely cancer" and the biopsy saying "uh, probably not?" that you need to decide what to do with.
  - hsuduebc2 2y ago
    I'm confused. If this blood test gave a false positive that wasn't due to an anomaly in the blood itself, then why can't we assume that the likelihood of getting a false positive twice is lower than getting it once?
    - mlyle 2y ago
      > I'm confused. If this blood test gave a false positive that wasn't due to an anomaly in the blood itself, That's the errant part. A small amount of the false positives will be because of lab issues. The rest will be because this patient is different in some way, but not all of them are cancer. Medicine doesn't have very many perfectly specific tests. So you have a patient who has some weird enzymes around because they're genetically different, for example, and they always pop positive on this particular test. Or has an unusual diet that causes some other non-tested-for-enzyme level to be high enough to set off this sensor. Or whatever. In this case, the specificity is 98%, so this false positive rate is about 2%.

caesil 2y ago

Then let's take those things into account when calculating what tests to do. Surely, though, we can do better as a society than solving this with "no preemptive testing except for extreme risks".

lm28469 2y ago
> Surely, though, we can do better as a society We haven't even solved the most basic shit like shelter, food, education, &c for millions of people in the west, as a society we're faaaaaaaaar from universal yearly full health checkups. As an individual feel free to get private checks, they'll gladly take your money
- aurizon 2y ago
  The fact that there are huge costs in the USA to even periodic medical checkups has severely impacted longevity in the USA to the point it ranks close to Cuba in longevity. Those with a health plan are close to the highest ranked nations. The poor without a plan at all are around ~4-5 less long lived. There is a nice rabbit hole in this data. https://www.google.com/search?q=longevity+charts&rlz=1C1CHZN_enCA1101CA1101&oq=longevity+charts&gs_lcrp=EgZjaHJvbWUyBggAEEUYOdIBCjE1MjMxajBqMTWoAgiwAgE&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=longevity+charts&rlz=1C1CHZN... This has a huge GDP cost in the USA, that needs to be addressed. The causes are big pharma/hospo/AMA/insuro/lobbyo.... One wonders why the AMA is there? - they limit the numbers of doctors trained in Universities/training hospitals to forestall price competition among doctors by various means. Dentists do the same.
  - toast0 2y ago
    Cost of a yearly checkup should be "taken care of", because Obamacare mandated free annual checkups, as long as you don't accidentally trigger any other billing codes while you're there. But, regardless of cost, there's a shortage of providers, so it's hard to schedule the checkup. And there's still a lot of uninsured people out there.
  - nradov 2y ago

Redoubts 2y ago

> You may cause lasting harm to some of those 105 people. Could you elaborate on this?

jtc331 2y ago
For example you might do surgery on people who wouldn’t benefit.
daedrdev 2y ago
you will do surgery on some of the 105 people. Some of them might die from complications, infections, etc or at least have lasting damage. Since several of the 5 people will not be any better off with treatment it's entirely possible that the screening produces palpably worse outcome. The earlier you screen, the worse this is, since the ratio of false positives vs true positives gets higher and higher, for example 1000 vs 5 or 10000 vs 5.
- tptacek 2y ago
  It's also psychologically harmful to have the positive test hanging over your head. I'm nearing the age where doctors start harassing about colonoscopies. You can do an at-home test instead of the full procedure, and it has a very good chance of ruling out the need for a colonoscopy. But it also has a high false positive rate; there's a decent chance that you'll end up in a state of "need a colonoscopy, also a colon cancer screener flagged you". I'm dreading the colonoscopy prep, but I'm not doing the at-home thing.
  - jeswin 2y ago
    I've had colonoscopies twice. Was just half a day of inconvenience if you schedule it for the morning.
    - tptacek 2y ago
      The scope, right? Not the at-home test? The only thing about the scope that bugs me is the prep.

AustinDev 2y ago

Can't you just run the test again instead of doing a full follow up? 1/1000 * 1/1000 = 1/1,000,000

okaram 2y ago
You only get that probability if the test results are completely uncorrelated, chances are, they're not. I'd assume the chances of getting a second false positive if you already got one are much higher.

tptacek 2y ago

For people surprised by this argument, the phenomenon he's invoking here has a name: the Bayesian Base Rate Fallacy.

fragmede 2y ago

> Society will pay for 100,000 tests For better or worse, under the American healthcare system, the patient pays for those tests, sometimes covered by insurance. If the tests are paid for out of pocket by the patient, is there still such issue?

s1artibartfast 2y ago
I think the issue is exactly the same no matter who pays. To reframe it from the individual patient's perspective, when you take a test simply for the sake of screening, there is the chance you'll learn something true that helps you, and the chance you will learn something false that hurts you
mlyle 2y ago
The economic argument doesn't change whether it's a private cost or purely a social cost (private costs are included in social costs, since private expenditures are part of society's expenditures).

fallingknife 2y ago

You are assuming that those false positive rates are fixed, but they aren't. The "positive" criteria are done by an analysis exactly as sophisticated as a human scanning a list of numbers. The process is a joke and it needs to be improved by more data and better analysis, not this nonsensical "don't test people because they might be positive" argument.

mlyle 2y ago
No, I'm assuming there's a tradeoff between sensitivity (spotting cancers) and specificity (having your positive results actually be cancer). ANOVA to pick variables and then reasonably selecting thresholds is a fine process that avoids overfit. The big problem is, biology is messy and measuring lots of people to find correct thresholds is really expensive and time consuming. It's not really a technological problem, though technology has helped a little.

hammock 2y ago

How come nobody seems uses this kind of math when it came to COVID prophylactics? Or did they?

dillydogg 2y ago

There was a seminar given to the breast cancer society by an epidemiologist years who who presented them with a scenario: Prevalence of breast cancer: 1% Sensitivity (percent of people who have the disease test positive): 90% Specificity (percent of people without the disease test negative): 91% And asked, "How many people who test positive have the disease?" (i.e., positive predictive value) It's only 1/11. I think only 20% got it right (in a 4 answer multiple choice question)

mlyle 2y ago
Yah, I'm well familiar with the base rate fallacy and I still catch myself screwing it up. It's so unintuitive. I use mental math a lot to counter my intuition. If sensitivity is high and the base rate is low, you can approximate it with .01 / (1-.91). Or, mental math assuming 10,000 people is not unreasonable for your case (100 with disease, 90 true positives; 9900 * 9/100 false positives.. divide everything by 9 to make it easier, 10 true positives and 99 false positives, or 10/109 or say "9%".