5 ms·

The MRI analogy is not good. The false positive risk is only against the present-day distribution of MRIs mostly taken of symptomatic patients; if we had the da

by sixo 1y ago

The MRI analogy is not good. The false positive risk is only against the present-day distribution of MRIs mostly taken of symptomatic patients; if we had the dataset of "annual MRIs for everyone" we would very quickly recalibrate our sensitivity to the new baseline.

causal 1y ago
False positives are still false positives. You don't decide to ignore a possible tumor because everybody's getting MRIs these days.
shazbotter 1y ago
MRIs produce shadows that are indistinguishable from cysts or tumors all the time. They are benign, but no amount of data will reduce them. And telling someone "you have a shadow here, it's probably benign" makes people anxious and they go, "maybe I should boost it?" Which is needlessly invasive.
- sixo 1y ago
  huh, well, if it's true that "no amount of data will reduce them" then my point is wrong, but I highly doubt that's true.
  - empyrrhicist 1y ago
    The data is often a biopsy, which is invasive and potentially harmful.
empyrrhicist 1y ago
... yeah? You'd expect the false positive rate to be HIGHER when you're not looking at an enriched patient subset. That's why we're careful about recommending certain kinds of screening. See also: PSA screening.
- sixo 1y ago
  Well, you missed my point. I'm not talking about "you look at the MRI and see something and say it's a positive", I'm referring to the process of reading MRIs as like a statistical model (even if in practice it exists in the minds of radiologists) which is trained on the corpus of MRI data. That model will depend in some way on the distribution of positive/negative examples in the corpus; if the corpus changes the model has to then be updated to match. Point is, the false positive concern is only a concern if you use the old model with the new corpus. Don't do that! That's dumb! The net effect of MRIing everyone on public health would likely be enormously positive as long as you don't do that.
  - empyrrhicist 1y ago
    Take PSA, since it's a simpler example. You're right that, if we screen everyone, taking action based on the outcome causes more harm than good. The response is to calibrate... which means we don't learn anything usefully actionable from the test and shouldn't apply it. With the MRI, you don't get back simple dichotomous things, but you get back potential indications. That can be scary - talk about calibration all you want, but if patients see things and start thinking about the big C word there are likely to be a lot of unnecessary biopsies. The bottom line is that it's possible to imagine a benefit, but it is not reasonable to pretend it's as simple as "just re-calibrate your interpretation of the results!". There's a reason that a lot of thought goes into when to do screening.
    - sixo 1y ago
      > which means we don't learn anything usefully actionable from the test and shouldn't apply it. This just isn't true. In practice any such screening model can ALWAYS improve with more data—basically because the statistical power goes up and up—up to an asymptote set by noise in the physical process itself. > That can be scary Handling that is the job of professionals, is now and will continue to be. It is extremely reasonable to imagine a benefit! What is doubtful is imagining there wouldn't be one! I find the line of reasoning in this whole anti-MRI-everyone argument to be bewildering. I think it is basically an emotional argument, which has set in as "established truth" by repetition; people will trot it out by instinct whenever they encounter any situation that suggests it. It reflects lessons collectively learned from the history of medicine, its over-estimation of its own abilities and its overfitting to data, and its ever-increasing sensitivity to liability. But it is not inherently true—it is really a statement about poor statistical and policy practices in the field, which could be rectified with concerted effort, with a potential for great public upside. Not that any of this matters at the current price point. But, on a brief investigation, the amortized cost of a single MRI scan is ~$500-800—perhaps 1/5 what I would have guessed!

by sixo 1y ago

causal 1y ago

False positives are still false positives. You don't decide to ignore a possible tumor because everybody's getting MRIs these days.

shazbotter 1y ago

MRIs produce shadows that are indistinguishable from cysts or tumors all the time. They are benign, but no amount of data will reduce them. And telling someone "you have a shadow here, it's probably benign" makes people anxious and they go, "maybe I should boost it?" Which is needlessly invasive.

sixo 1y ago
huh, well, if it's true that "no amount of data will reduce them" then my point is wrong, but I highly doubt that's true.
- empyrrhicist 1y ago
  The data is often a biopsy, which is invasive and potentially harmful.

empyrrhicist 1y ago

... yeah? You'd expect the false positive rate to be HIGHER when you're not looking at an enriched patient subset. That's why we're careful about recommending certain kinds of screening. See also: PSA screening.

sixo 1y ago
Well, you missed my point. I'm not talking about "you look at the MRI and see something and say it's a positive", I'm referring to the process of reading MRIs as like a statistical model (even if in practice it exists in the minds of radiologists) which is trained on the corpus of MRI data. That model will depend in some way on the distribution of positive/negative examples in the corpus; if the corpus changes the model has to then be updated to match. Point is, the false positive concern is only a concern if you use the old model with the new corpus. Don't do that! That's dumb! The net effect of MRIing everyone on public health would likely be enormously positive as long as you don't do that.
- empyrrhicist 1y ago
  Take PSA, since it's a simpler example. You're right that, if we screen everyone, taking action based on the outcome causes more harm than good. The response is to calibrate... which means we don't learn anything usefully actionable from the test and shouldn't apply it. With the MRI, you don't get back simple dichotomous things, but you get back potential indications. That can be scary - talk about calibration all you want, but if patients see things and start thinking about the big C word there are likely to be a lot of unnecessary biopsies. The bottom line is that it's possible to imagine a benefit, but it is not reasonable to pretend it's as simple as "just re-calibrate your interpretation of the results!". There's a reason that a lot of thought goes into when to do screening.
  - sixo 1y ago
    > which means we don't learn anything usefully actionable from the test and shouldn't apply it. This just isn't true. In practice any such screening model can ALWAYS improve with more data—basically because the statistical power goes up and up—up to an asymptote set by noise in the physical process itself. > That can be scary Handling that is the job of professionals, is now and will continue to be. It is extremely reasonable to imagine a benefit! What is doubtful is imagining there wouldn't be one! I find the line of reasoning in this whole anti-MRI-everyone argument to be bewildering. I think it is basically an emotional argument, which has set in as "established truth" by repetition; people will trot it out by instinct whenever they encounter any situation that suggests it. It reflects lessons collectively learned from the history of medicine, its over-estimation of its own abilities and its overfitting to data, and its ever-increasing sensitivity to liability. But it is not inherently true—it is really a statement about poor statistical and policy practices in the field, which could be rectified with concerted effort, with a potential for great public upside. Not that any of this matters at the current price point. But, on a brief investigation, the amortized cost of a single MRI scan is ~$500-800—perhaps 1/5 what I would have guessed!