16 ms·

> Balzer used Photoshop to layer the new DICOM files on top of the old images, and realized that the tumor hadn’t grown at all — the radiologist had just measur

> Balzer used Photoshop to layer the new DICOM files on top of the old images, and realized that the tumor hadn’t grown at all — the radiologist had just measured from a different point on the image. Think about the some of the implications of that statement for a while. It really is no wonder that the softer sciences have a reproducibility problem.

drdeadringer 12y ago
> the radiologist had just measured from a different point on the image. I had a similar experience when I was having a spinal curvature corrected. After a particular xray, several people -- a couple doctors, the xray guy, a med student or two, a mix of a couple people like this -- had measured my spine's then-current curvature and had come up with a huge range; to one, I had gotten worse... to another, I had some improvement... to the third, marginally the same with a smudge of improvement. In the end I got the most improvement I was going to get given my specific situation and I'm happy for//with that. The point is, it was literally three measurements on the same xray from which three mutually exclusive conclusions//opinions were [literally?] drawn. I'm therefore not surprised that there is similar with tumours.
VikingCoder 12y ago
There are multiple problems. The biggest problem in my mind is that outcomes are not tracked well. That means that there's no medical feedback loop to measure the long-term effectiveness of... anything. And no good way to do retrospective studies to prove new measurement techniques. And also the laws for patient protection makes it incredibly difficult to get even anonymized data for researchers to use. The next biggest problem is that there's a gold standard of treatment, and it's hard to get enough research to move the gold standard. In this case, the gold standard is essentially a diameter measurement, based on picking the visibly largest slice of a data set which happens to have an axis determined by the patient's somewhat random orientation in the CT scanner. Axial, Coronal, Sagittal. Those are not perfect, anatomical orientations - they're rough. Most radiologists won't take the time to create Multi-Planar Reformats along oblique or double-oblique angles. And even if they did, they'd still be eye-balling the measurement. It's not very hard from a regulatory standpoint to create tools to allow a radiologist to click here, click there, and show a measurement of distance. Then the radiologist can look at a similar measurement from a different radiologist on the previous scan, and say whether it's bigger or smaller now. (But some PACS still don't handle anisotropic MPR correctly, which just blows my mind.) It's vastly harder from a regulatory standpoint to say in a medical product for commercial use, "Hey, we automatically measured the volume of this tumor by automatically finding the boundary of the tissue and automatically ignoring the connected blood vessels and neighboring bone, and then we automatically found the largest cross-sectional diameter, and then we automatically compared the same measurements against the automatically non-linearly-registered longitudinal previous scan... diagnosis: the tumor is not growing, just changing shape slightly." People think their CT scans are examined for hours by a team of doctors, like on "House." In reality, you're lucky if your CT scan gets 5 minutes of investigation, and that's by a radiologist who was trained in med school to read X-Rays, not CT. And your oncology head CT may be read by a radiologist who specializes in abdominal pediatric Ultrasound. And there's almost certainly no double-reading for Quality Assurance. And there's no rating on radiologists to tell you who has the most accurate diagnosis. And he's probably using 2D measuring tools on 3D data - because that's what they're familiar with, and that's what the reconstruction software manufacturers provide, and that's the medical gold standard. On a shared computer, where some other radiologist (or resident!) might have changed the display settings in a way he didn't notice. And he's probably been reading images for 14 hours straight.
- gmarx 12y ago
  Medicine has a data problem. It is multifaceted and far more difficult to fix than it should be. I've worked on these problems for years and it is immensely frustrating. Incentives just aren't set properly to solve them. For example with outcomes tracking, even if you had a good software system for it, you won't get reimbursed for it.
  - jpmattia 12y ago
    > For example with outcomes tracking, even if you had a good software system for it, you won't get reimbursed for it. It's worse than that: It would be sued out of existence by those people who rank badly (ie, exactly those bad practitioners that need to have their asses kicked.) In a related example: A friend wrote a book about first-hand experience in dealing with a nursing home for her aging parent. There were many instances of problems, but one in particular was that a doc had prescribed the wrong drug. The publisher loved the book, and made the publication offer contingent on a number of changes. One of those changes: You can't implicate a person or his institution because of the liability. It doesn't matter how good your proof is, the jury system is so broken that the publisher just can't accept the risk. Now just imagine if there were a searchable database of outcomes tied to individual practitioners or institutions.
    - VikingCoder 12y ago
      You're not going quite far enough - Medical Practitioners and Medical Institutions should be required by law to document outcomes, provide information about disputed outcomes, and update the information on a periodic basis. And tampering with the information should carry heavy punishment, including jail time for the practitioner and officers of the institution. Actions like the recent cover-ups in the VA hospitals about wait times leading to deaths should be nearly impossible to hide from the public eye.
      - gmarx 12y ago
        You'd need to take account of differences in patients lest you discourage doctors from treating the more unhealthy. You also need to alter the way we define and punish malpractice.
nhstanley 12y ago
> It really is no wonder that the softer sciences have a reproducibility problem. Even Further, I think it's also indicative of medicine being full of people that have a fatalistic, passive attitude towards their patients and medical care. Medicine and health is really hard, and so many health professionals essentially believe that the only thing they can do is wait until something horrible happens before doing anything. And as the case with this woman, when something bad does happen, they bin it in their per-organized mental filing cabinet and you're fucked if they're wrong. Much like police officers, I think many get so jaded by the job that they can't be bothered to give a shit when they really should. And I really hope technology will save us from this.
- collyw 12y ago
  Like most professions, I am sure there are doctors who are like you describe as well as many others at they opposite end of the spectrum. Its wrong to generalize them all to be like this.
- amedstudent1 12y ago
  I think the other reply covers it a lot, but I find posts like yours to be very demeaning. You're essentially doing the same as someone who doesn't understand programming making declarations of how programs should be made. There is a lot going on behind the scenes and modern medicine is based on evidence, so please take the arm-chair commentary away
- tjradcliffe 12y ago
  It's somewhat less depressing than that, from a personal point of view, at least. Most interventions carry with them pretty significant risk. Therefore the choice is not between "do nothing and see what happens, or do something with zero risk and see if it helps", but rather "do nothing and see what happens, or do something really high risk and see if it does more harm than good." Iatrogenic disease (harm caused by interventions) is by some measures the leading cause of death in the US, outstripping heart disease and standing at more than ten times the rate of death by automobile. Under those circumstances, not doing anything until something horrible happens is not a bad policy, and the people engaged in it are not expressing cynical indifference so much as a due awareness of the real risks involved in almost any intervention. Which is, admittedly, kind of depressing, but for quite different reasons. Either way, technology and patient empowerment are likely to help.
romaniv 12y ago
Think about the some of the implications of that statement for a while. Hm, people with 6-digit salaries and doctoral degrees whose main job is to figure out diseases perform worse than some random dude? That's the first thing that comes to my mind.
neurotech1 12y ago
"Trust but verify": There are standards for tumor measurement. However, for something with major life-altering consequences, having the same radiologist compare scans is important to avoid observational variability. A neurologist I know, she always checks the actual scans as opposed to relying on the report narrative alone. Her background is in computer science, before her MD. The issue with new analysis tools is that most doctors won't stake their reputation by exclusion. If a doctor says the tumor isn't growing, and it is, they are liable. If a neurosurgeon removes a benign tumor, then its business as usual.
Shinkei 12y ago
Softer sciences? It was simple human error, not a failing of the science itself. I've made a more extensive post separately, but in short... regulations have slowed the advances that could prevent this kind of human error.
caycep 12y ago
Most research labs can do something a little fancier but basically similar, using mutual information reduction algorithms to mathematically align images. The software is also open source (see ANTS from UPenn or FSL from Oxford). Sadly, none of this is easily available in FDA approved packages from typical MRI/CT imaging vendors like GE or Siemens. Often need to pay extra for such features. You could theoretically walk to the surgery suites and borrow a stereotactic imaging station (basically an embedded Linux PC) and do it there, but radiologists typically aren't trained to do that. FSL and ANTS aren't approved by FDA for clinical use so can't be officially documented in a patient's chart... Most radiology training is based on visual recognition of abnormalities in anatomy - the "eyeball approach" has its value, which is why intelligence agencies employ human spotters and people are working on neuromorphic chips. But there isn't really a lot of quantitative/comp sci expertise among typical medical students (maybe more now, who knows?)

drdeadringer 12y ago

> the radiologist had just measured from a different point on the image. I had a similar experience when I was having a spinal curvature corrected. After a particular xray, several people -- a couple doctors, the xray guy, a med student or two, a mix of a couple people like this -- had measured my spine's then-current curvature and had come up with a huge range; to one, I had gotten worse... to another, I had some improvement... to the third, marginally the same with a smudge of improvement. In the end I got the most improvement I was going to get given my specific situation and I'm happy for//with that. The point is, it was literally three measurements on the same xray from which three mutually exclusive conclusions//opinions were [literally?] drawn. I'm therefore not surprised that there is similar with tumours.

VikingCoder 12y ago

There are multiple problems. The biggest problem in my mind is that outcomes are not tracked well. That means that there's no medical feedback loop to measure the long-term effectiveness of... anything. And no good way to do retrospective studies to prove new measurement techniques. And also the laws for patient protection makes it incredibly difficult to get even anonymized data for researchers to use. The next biggest problem is that there's a gold standard of treatment, and it's hard to get enough research to move the gold standard. In this case, the gold standard is essentially a diameter measurement, based on picking the visibly largest slice of a data set which happens to have an axis determined by the patient's somewhat random orientation in the CT scanner. Axial, Coronal, Sagittal. Those are not perfect, anatomical orientations - they're rough. Most radiologists won't take the time to create Multi-Planar Reformats along oblique or double-oblique angles. And even if they did, they'd still be eye-balling the measurement. It's not very hard from a regulatory standpoint to create tools to allow a radiologist to click here, click there, and show a measurement of distance. Then the radiologist can look at a similar measurement from a different radiologist on the previous scan, and say whether it's bigger or smaller now. (But some PACS still don't handle anisotropic MPR correctly, which just blows my mind.) It's vastly harder from a regulatory standpoint to say in a medical product for commercial use, "Hey, we automatically measured the volume of this tumor by automatically finding the boundary of the tissue and automatically ignoring the connected blood vessels and neighboring bone, and then we automatically found the largest cross-sectional diameter, and then we automatically compared the same measurements against the automatically non-linearly-registered longitudinal previous scan... diagnosis: the tumor is not growing, just changing shape slightly." People think their CT scans are examined for hours by a team of doctors, like on "House." In reality, you're lucky if your CT scan gets 5 minutes of investigation, and that's by a radiologist who was trained in med school to read X-Rays, not CT. And your oncology head CT may be read by a radiologist who specializes in abdominal pediatric Ultrasound. And there's almost certainly no double-reading for Quality Assurance. And there's no rating on radiologists to tell you who has the most accurate diagnosis. And he's probably using 2D measuring tools on 3D data - because that's what they're familiar with, and that's what the reconstruction software manufacturers provide, and that's the medical gold standard. On a shared computer, where some other radiologist (or resident!) might have changed the display settings in a way he didn't notice. And he's probably been reading images for 14 hours straight.

gmarx 12y ago
Medicine has a data problem. It is multifaceted and far more difficult to fix than it should be. I've worked on these problems for years and it is immensely frustrating. Incentives just aren't set properly to solve them. For example with outcomes tracking, even if you had a good software system for it, you won't get reimbursed for it.
- jpmattia 12y ago
  > For example with outcomes tracking, even if you had a good software system for it, you won't get reimbursed for it. It's worse than that: It would be sued out of existence by those people who rank badly (ie, exactly those bad practitioners that need to have their asses kicked.) In a related example: A friend wrote a book about first-hand experience in dealing with a nursing home for her aging parent. There were many instances of problems, but one in particular was that a doc had prescribed the wrong drug. The publisher loved the book, and made the publication offer contingent on a number of changes. One of those changes: You can't implicate a person or his institution because of the liability. It doesn't matter how good your proof is, the jury system is so broken that the publisher just can't accept the risk. Now just imagine if there were a searchable database of outcomes tied to individual practitioners or institutions.
  - VikingCoder 12y ago
    You're not going quite far enough - Medical Practitioners and Medical Institutions should be required by law to document outcomes, provide information about disputed outcomes, and update the information on a periodic basis. And tampering with the information should carry heavy punishment, including jail time for the practitioner and officers of the institution. Actions like the recent cover-ups in the VA hospitals about wait times leading to deaths should be nearly impossible to hide from the public eye.
    - gmarx 12y ago
      You'd need to take account of differences in patients lest you discourage doctors from treating the more unhealthy. You also need to alter the way we define and punish malpractice.

nhstanley 12y ago

> It really is no wonder that the softer sciences have a reproducibility problem. Even Further, I think it's also indicative of medicine being full of people that have a fatalistic, passive attitude towards their patients and medical care. Medicine and health is really hard, and so many health professionals essentially believe that the only thing they can do is wait until something horrible happens before doing anything. And as the case with this woman, when something bad does happen, they bin it in their per-organized mental filing cabinet and you're fucked if they're wrong. Much like police officers, I think many get so jaded by the job that they can't be bothered to give a shit when they really should. And I really hope technology will save us from this.

collyw 12y ago
Like most professions, I am sure there are doctors who are like you describe as well as many others at they opposite end of the spectrum. Its wrong to generalize them all to be like this.
amedstudent1 12y ago
I think the other reply covers it a lot, but I find posts like yours to be very demeaning. You're essentially doing the same as someone who doesn't understand programming making declarations of how programs should be made. There is a lot going on behind the scenes and modern medicine is based on evidence, so please take the arm-chair commentary away
tjradcliffe 12y ago
It's somewhat less depressing than that, from a personal point of view, at least. Most interventions carry with them pretty significant risk. Therefore the choice is not between "do nothing and see what happens, or do something with zero risk and see if it helps", but rather "do nothing and see what happens, or do something really high risk and see if it does more harm than good." Iatrogenic disease (harm caused by interventions) is by some measures the leading cause of death in the US, outstripping heart disease and standing at more than ten times the rate of death by automobile. Under those circumstances, not doing anything until something horrible happens is not a bad policy, and the people engaged in it are not expressing cynical indifference so much as a due awareness of the real risks involved in almost any intervention. Which is, admittedly, kind of depressing, but for quite different reasons. Either way, technology and patient empowerment are likely to help.

romaniv 12y ago

Think about the some of the implications of that statement for a while. Hm, people with 6-digit salaries and doctoral degrees whose main job is to figure out diseases perform worse than some random dude? That's the first thing that comes to my mind.

neurotech1 12y ago

"Trust but verify": There are standards for tumor measurement. However, for something with major life-altering consequences, having the same radiologist compare scans is important to avoid observational variability. A neurologist I know, she always checks the actual scans as opposed to relying on the report narrative alone. Her background is in computer science, before her MD. The issue with new analysis tools is that most doctors won't stake their reputation by exclusion. If a doctor says the tumor isn't growing, and it is, they are liable. If a neurosurgeon removes a benign tumor, then its business as usual.

Shinkei 12y ago

Softer sciences? It was simple human error, not a failing of the science itself. I've made a more extensive post separately, but in short... regulations have slowed the advances that could prevent this kind of human error.

caycep 12y ago

Most research labs can do something a little fancier but basically similar, using mutual information reduction algorithms to mathematically align images. The software is also open source (see ANTS from UPenn or FSL from Oxford). Sadly, none of this is easily available in FDA approved packages from typical MRI/CT imaging vendors like GE or Siemens. Often need to pay extra for such features. You could theoretically walk to the surgery suites and borrow a stereotactic imaging station (basically an embedded Linux PC) and do it there, but radiologists typically aren't trained to do that. FSL and ANTS aren't approved by FDA for clinical use so can't be officially documented in a patient's chart... Most radiology training is based on visual recognition of abnormalities in anatomy - the "eyeball approach" has its value, which is why intelligence agencies employ human spotters and people are working on neuromorphic chips. But there isn't really a lot of quantitative/comp sci expertise among typical medical students (maybe more now, who knows?)