4 ms·

I coauthored a paper a few years ago on the intra-observer variability of measurements of brain tumors when using the bidimensional product. An increase of 25%

I coauthored a paper a few years ago on the intra-observer variability of measurements of brain tumors when using the bidimensional product. An increase of 25% in the BP is taken as an indicator of brain tumor progression. We found that the intra-observer variability is so high, you can get that 25% increase purely by chance. The first recommendation of the medical team, to wait and see, is understandable.

escape_goat 12y ago
From your wording, it seems like your brain might have slipped gears when talking about probabilities, from 'science' mode to 'homo sapiens sapiens hardwired quantity estimator' mode. When you say that "you can get that 25% increase purely by chance" do you mean (a) in better circumstances this should be formally impossible, or (b) the intra-observer variability is so high that there is significant (by some definition) possibility ___ of falsely detecting a 25% increase?
billiam 12y ago
That is a shocking thing to say. If they had followed the wait and see approach there's a good chance she might be blind right now. This high-handedness is why I have such a hard time with the medical profession.
- gwern 12y ago
  > If they had followed the wait and see approach there's a good chance she might be blind right now. Operating on a brain unnecessarily has its own risks.
  - kbenson 12y ago
    Indeed, as someone who's had a close family member require brain surgery in the past, I would caution people against thinking of it like any other surgery. The equivalent of "a lingering stiffness" or "Skin tightening because of scar tissue" becomes something else entirely when talking about the brain, and by extension the mind.
- zo1 12y ago
  And you don't think it may have something to do with the amount of lawsuits that is open to them if they make a mistake? We can't claim that the thought/idea of a lawsuit over a medical mistake looming is always 0, because it isn't. Even on a sub-conscious level, it affects and weighs in on the medical practitioner's diagnoses.
- mcmancini 12y ago
  Hospitals can be a pretty dangerous place. You're a braver person than I if you're willing to treat a likely benign and possibly asymptomatic meningioma without fully investigating the lesion, including rate of progression.
tjradcliffe 12y ago
I'm kind of fascinated that this sort of thing is still considered a publishable result. It's been known for at least twenty years that intra-observer variability is a significant factor in such things. I'm not knocking the work you've done, but pointing out that when I was working in radiotherapy in the early 90's this kind of stuff (not on bidimensional product specifically, but whatever tumor volume measures were being used at the time) was being done. I wasn't involved in any of it directly, but recall colleagues who were, more-or-less ranting on about the degree to which chance predominated. One issue I was more involved in was that every modality (CT, MR, fluoro...) gave significantly different tumour volumes and no one knew how to reconcile them in a clinically useful way. Given that every measure we've ever devised seems subject to similar issues, there's an argument to be made that the very concept of "tumour size" is problematic. It seems to make sense, but the edges are too fuzzy to be very diagnostically useful in a depressingly large number of cases.
- mcmancini 12y ago
  Yeah, I was surprised too when the project was pitched to me. At the time we did the article, a working group had proposed a revision to the Macdonald criteria that still included bidimensional products, and this paper was in part a response to the revision. I think that's why our article was interesting enough for publication. This area is really neat to me because it seems at first to be easy: just take a couple of pictures, compare them, and there's the answer, right? But then you start thinking about differences in imaging modalities, contrast agents, calibration, signal processing, measurement axis, etc. and the problem becomes so hard I want to find something easier to work on.
pjungwir 12y ago
Having been to three ultrasounds for my children, I'm amazed how casually they use a 2D length to estimate the baby's age, and therefore delivery date, and therefore when it is overdue and calls for a cesarean, etc. Your results seem like they confirm my suspicions that this can't possibly be very accurate.
- Stefan-H 12y ago
  With babies we know the shape. It is not as if we are likely to get a 2d slice of just an arm to estimate the age. With tumors we do not have an idea of the shape, and therefore could accidentally take a slice of the narrowest portion the first time, then the largest portion the second and assume it grew.
- snowwrestler 12y ago
  The real problem here is the concept of "overdue" tied to interventions. Babies are like wizards: they are never late. They arrive promptly when they intend to. And there is actually not much evidence that arriving within a week or two on either side of the "due date" has any correlation with health problems for the mother or baby. So, a well-educated OB can be a bit casual about the exact age of the fetus because it doesn't really matter all that much.
- Florin_Andrei 12y ago
  Or maybe you're seeing just a slice of a procedure with a more complex theory behind it that you're not aware of. Remember, being good at computers does not imply being good at medicine. This should be self-evident, but just a quick perusal of some comments here shows the contrary.
  - pjungwir 12y ago
    Yes, I completely agree. It's not like I'd take action on my doubts. I'm sorry if my original comment sounded critical---that wasn't my point. But it is crude, isn't it? I believe even doctors concede that these estimates are not very accurate (especially when taken mid/late in the pregnancy), and it is sort of alarming to watch a nurse snap lines across the skull cross-section freehand and use those. I find that in general with medicine my intuition battles against what my intellect knows must be the case. It's not like anything in the body is far away. Cancer isn't even very small. Isn't it weird that it's so hard to treat? Obviously it must be, but it feels like it should be easy. The patient is a foot away. We are inscrutably powerless. When every day you command computers and they obey, when you can summon distant people and their image appears before you, when you can instantly consult more lore than in the greatest library, when you see people casually toting radiant tablets of spellbinding powers, progress in medicine feels so slow, so analog---so carnal. I know it's wrong, and we live in a world of medical miracles, with vaccines and antibiotics and DNA and everything else, but sometimes I can't help but feel that nothing has happened since Galen. It's not an argument, it's a feeling, but one with pathos that I'm sure many share.
mzs 12y ago
A terrible aspect to this is that the criteria from utilization management often list a minimum tumor size or showing growth and that same technique is used.

escape_goat 12y ago

From your wording, it seems like your brain might have slipped gears when talking about probabilities, from 'science' mode to 'homo sapiens sapiens hardwired quantity estimator' mode. When you say that "you can get that 25% increase purely by chance" do you mean (a) in better circumstances this should be formally impossible, or (b) the intra-observer variability is so high that there is significant (by some definition) possibility ___ of falsely detecting a 25% increase?

billiam 12y ago

That is a shocking thing to say. If they had followed the wait and see approach there's a good chance she might be blind right now. This high-handedness is why I have such a hard time with the medical profession.

gwern 12y ago
> If they had followed the wait and see approach there's a good chance she might be blind right now. Operating on a brain unnecessarily has its own risks.
- kbenson 12y ago
  Indeed, as someone who's had a close family member require brain surgery in the past, I would caution people against thinking of it like any other surgery. The equivalent of "a lingering stiffness" or "Skin tightening because of scar tissue" becomes something else entirely when talking about the brain, and by extension the mind.
zo1 12y ago
And you don't think it may have something to do with the amount of lawsuits that is open to them if they make a mistake? We can't claim that the thought/idea of a lawsuit over a medical mistake looming is always 0, because it isn't. Even on a sub-conscious level, it affects and weighs in on the medical practitioner's diagnoses.
mcmancini 12y ago
Hospitals can be a pretty dangerous place. You're a braver person than I if you're willing to treat a likely benign and possibly asymptomatic meningioma without fully investigating the lesion, including rate of progression.

tjradcliffe 12y ago

I'm kind of fascinated that this sort of thing is still considered a publishable result. It's been known for at least twenty years that intra-observer variability is a significant factor in such things. I'm not knocking the work you've done, but pointing out that when I was working in radiotherapy in the early 90's this kind of stuff (not on bidimensional product specifically, but whatever tumor volume measures were being used at the time) was being done. I wasn't involved in any of it directly, but recall colleagues who were, more-or-less ranting on about the degree to which chance predominated. One issue I was more involved in was that every modality (CT, MR, fluoro...) gave significantly different tumour volumes and no one knew how to reconcile them in a clinically useful way. Given that every measure we've ever devised seems subject to similar issues, there's an argument to be made that the very concept of "tumour size" is problematic. It seems to make sense, but the edges are too fuzzy to be very diagnostically useful in a depressingly large number of cases.

mcmancini 12y ago
Yeah, I was surprised too when the project was pitched to me. At the time we did the article, a working group had proposed a revision to the Macdonald criteria that still included bidimensional products, and this paper was in part a response to the revision. I think that's why our article was interesting enough for publication. This area is really neat to me because it seems at first to be easy: just take a couple of pictures, compare them, and there's the answer, right? But then you start thinking about differences in imaging modalities, contrast agents, calibration, signal processing, measurement axis, etc. and the problem becomes so hard I want to find something easier to work on.

pjungwir 12y ago

Having been to three ultrasounds for my children, I'm amazed how casually they use a 2D length to estimate the baby's age, and therefore delivery date, and therefore when it is overdue and calls for a cesarean, etc. Your results seem like they confirm my suspicions that this can't possibly be very accurate.

Stefan-H 12y ago
With babies we know the shape. It is not as if we are likely to get a 2d slice of just an arm to estimate the age. With tumors we do not have an idea of the shape, and therefore could accidentally take a slice of the narrowest portion the first time, then the largest portion the second and assume it grew.
snowwrestler 12y ago
The real problem here is the concept of "overdue" tied to interventions. Babies are like wizards: they are never late. They arrive promptly when they intend to. And there is actually not much evidence that arriving within a week or two on either side of the "due date" has any correlation with health problems for the mother or baby. So, a well-educated OB can be a bit casual about the exact age of the fetus because it doesn't really matter all that much.
Florin_Andrei 12y ago
Or maybe you're seeing just a slice of a procedure with a more complex theory behind it that you're not aware of. Remember, being good at computers does not imply being good at medicine. This should be self-evident, but just a quick perusal of some comments here shows the contrary.
- pjungwir 12y ago
  Yes, I completely agree. It's not like I'd take action on my doubts. I'm sorry if my original comment sounded critical---that wasn't my point. But it is crude, isn't it? I believe even doctors concede that these estimates are not very accurate (especially when taken mid/late in the pregnancy), and it is sort of alarming to watch a nurse snap lines across the skull cross-section freehand and use those. I find that in general with medicine my intuition battles against what my intellect knows must be the case. It's not like anything in the body is far away. Cancer isn't even very small. Isn't it weird that it's so hard to treat? Obviously it must be, but it feels like it should be easy. The patient is a foot away. We are inscrutably powerless. When every day you command computers and they obey, when you can summon distant people and their image appears before you, when you can instantly consult more lore than in the greatest library, when you see people casually toting radiant tablets of spellbinding powers, progress in medicine feels so slow, so analog---so carnal. I know it's wrong, and we live in a world of medical miracles, with vaccines and antibiotics and DNA and everything else, but sometimes I can't help but feel that nothing has happened since Galen. It's not an argument, it's a feeling, but one with pathos that I'm sure many share.

mzs 12y ago

A terrible aspect to this is that the criteria from utilization management often list a minimum tumor size or showing growth and that same technique is used.