6 ms·

> I often think about how we lack truely objective assessment of patient outcomes We don't really need truely objective assessments. As an analogy, imagine w

by MarkMc 9y ago

> I often think about how we lack truely objective assessment of patient outcomes We don't really need truely objective assessments. As an analogy, imagine we randomly added or subtracted 3 minutes to the finishing times of all the runners in the Boston marathon. We wouldn't have a truely objective assessment of each runner, but it's still a useful guide about who is likely to beat whom in the next marathon. > - Each patient's case is unique and cases with the highest levels of difficulty are often treated by the most experienced people. These cases, of course, are likely to have worse outcomes than simple cases which may be treated by less experienced (worse?) physicians. This problem can be alleviated to a large degree by risk-adjusting the outcomes: For each patient, estimate the most likely outcome and compare the actual outcome to the estimate. > - Clinical outcomes are largely recorded by the same people treating the patient so reported outcomes are often erroneous or frankly fraudulent. I don't see this as an insurmountable obstacle. For example, you could have an independent body randomly sample the reported outcomes to check that they are accurate, and apply some kind of penalty if they are not.

neuro_imager 9y ago
> We don't really need truely objective assessments. As an analogy, imagine we randomly added or subtracted 3 minutes to the finishing times of all the runners in the Boston marathon. We wouldn't have a truely objective assessment of each runner, but it's still a useful guide about who is likely to beat whom in the next marathon. I don't think the analogy fits. There's no single metric (like runner speed/time) to evaluate competence. > This problem can be alleviated to a large degree by risk-adjusting the outcomes: For each patient, estimate the most likely outcome and compare the actual outcome to the estimate. Agree. > I don't see this as an insurmountable obstacle. For example, you could have an independent body randomly sample the reported outcomes to check that they are accurate, and apply some kind of penalty if they are not. That would be a good solution. Part of the problem is finding a truely independent body. Medicine is over-run with various governance and regulatory bodies (several of which have been shown to be little more than rent collectors). And again, there's the problem of deference to eminence combined with small in-bred communities within each speciality or sub-specialty (who would likely be the only people with the training to evaluate their peers reliably).
- MarkMc 9y ago
  > I don't think the analogy fits. There's no single metric (like runner speed/time) to evaluate competence. Let me suggest such a metric: A T-value. That is, the degree to which the actual outcome deviates from the expected outcome. For example, say you have a hospital that does stem cell transplants. For each patient before the treatment you assess the "chance that the patient will die within 1 year" based on that patient's age, sex, BMI, heart and lung function, type of disease, time since last relapse, quality of donor match, etc. From this you estimate that 19.6% of the hospital's patients who are treated over a particular period will die within 1 year, with a standard deviation of 3.7%. The actual mortality rate for this group of patients turns out to be 26.2%. So the T-value is +1.78; this is the 'single metric' used to evaluate the competence of the hospital. > there's the problem of deference to eminence combined with small in-bred communities within each speciality or sub-specialty (who would likely be the only people with the training to evaluate their peers reliably). Keep in mind that expertise is only required in estimating pre-treatment "chance that the patient will die within 1 year". Determining whether a patient is alive after a year and calculating the T-value can be done by anyone. And even when estimating expected mortality, there is plenty of evidence that a simple algorithm can actually beat the experts - Daniel Kahneman devotes a whole chapter to this point in 'Thinking Fast and Slow' [1]. [1] Chapter 21: https://www.amazon.com/dp/B00555X8OA/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1 https://www.amazon.com/dp/B00555X8OA/ref=dp-kindle-redirect?...
  - neuro_imager 9y ago
    Overall, I like the idea of a metric such as the one you describe. However, a change in one year mortality rate is unlikely to be a reliable indicator of the efficiency of most treatments. Most medical interventions have more modest effects than definite improvement in short term survival.
    - MarkMc 9y ago
      Sure, there might be better metrics than 1-year survival and different treatments could have different metrics.
nradov 9y ago
Nice theory, but in reality there's no way to do accurate risk adjustment. We just don't have enough data and so many physicians would end up unfairly penalized due to factors outside their control. Perhaps in 100 years we'll be in a better position to do something like that.
- tinokid 9y ago
  Is this an argument against tracking and releasing data? Let people do their own risk adjustments. If twice as many people who are going to one doctor or hospital are ending up dead or crippled, and there are no discernible confounding factors (age, socioeconomic, co-morbidities, etc), then I want to know. One might even say the patient has a right to know. The burden should be on the provider/facility to convince the patient to trust them nonetheless.
  - nradov 9y ago
    Yes this is an argument against releasing data. You and most other patients lack the skills and context to interpret the data in a meaningful way. And no provider currently captures all of the confounding factors. In fact, for most serious conditions we haven't even done enough research to understand what all the confounding factors are. If we go with your proposal then the inevitable outcome is that the best providers (particularly surgeons) will engage in metrics arbitrage by refusing to treat patients whose co-morbidities and complications aren't adequately captured by standardized coding systems and clinical guidelines. Is that really the outcome you want? As for putting the burden on providers to convince patients to trust them, good luck with that. We currently have shortages of providers in many areas and specialties due to price fixing and supply constraints. So most patients have to take what they can get regardless of trust.
    - tinokid 9y ago
      Your views on bodily autonomy seem bizarre. Patients own their own bodies and have the right to make fully informed decisions about what happens to them. If a provider's or facility's (or entire speciality's) numbers look bad, that's important information for the consumer to possess. Hiding the numbers should not be an option. Or do you think that car crash test results, and airplane crash data, and health department inspection findings should also be kept secret? More problematically, if you can't articulate a quantifiable standard that will indicate a provider's quality level, then then entire concept seems ill-defined. I.e., when you refer to the "best providers", what does that even mean?
adrianN 9y ago
The problem is not randomly adding a couple of minutes that you can average out again, but systemic biases like the one your parent mentioned: difficult cases are treated by different doctors than easier cases.

by MarkMc 9y ago

neuro_imager 9y ago

> We don't really need truely objective assessments. As an analogy, imagine we randomly added or subtracted 3 minutes to the finishing times of all the runners in the Boston marathon. We wouldn't have a truely objective assessment of each runner, but it's still a useful guide about who is likely to beat whom in the next marathon. I don't think the analogy fits. There's no single metric (like runner speed/time) to evaluate competence. > This problem can be alleviated to a large degree by risk-adjusting the outcomes: For each patient, estimate the most likely outcome and compare the actual outcome to the estimate. Agree. > I don't see this as an insurmountable obstacle. For example, you could have an independent body randomly sample the reported outcomes to check that they are accurate, and apply some kind of penalty if they are not. That would be a good solution. Part of the problem is finding a truely independent body. Medicine is over-run with various governance and regulatory bodies (several of which have been shown to be little more than rent collectors). And again, there's the problem of deference to eminence combined with small in-bred communities within each speciality or sub-specialty (who would likely be the only people with the training to evaluate their peers reliably).

MarkMc 9y ago
> I don't think the analogy fits. There's no single metric (like runner speed/time) to evaluate competence. Let me suggest such a metric: A T-value. That is, the degree to which the actual outcome deviates from the expected outcome. For example, say you have a hospital that does stem cell transplants. For each patient before the treatment you assess the "chance that the patient will die within 1 year" based on that patient's age, sex, BMI, heart and lung function, type of disease, time since last relapse, quality of donor match, etc. From this you estimate that 19.6% of the hospital's patients who are treated over a particular period will die within 1 year, with a standard deviation of 3.7%. The actual mortality rate for this group of patients turns out to be 26.2%. So the T-value is +1.78; this is the 'single metric' used to evaluate the competence of the hospital. > there's the problem of deference to eminence combined with small in-bred communities within each speciality or sub-specialty (who would likely be the only people with the training to evaluate their peers reliably). Keep in mind that expertise is only required in estimating pre-treatment "chance that the patient will die within 1 year". Determining whether a patient is alive after a year and calculating the T-value can be done by anyone. And even when estimating expected mortality, there is plenty of evidence that a simple algorithm can actually beat the experts - Daniel Kahneman devotes a whole chapter to this point in 'Thinking Fast and Slow' [1]. [1] Chapter 21: https://www.amazon.com/dp/B00555X8OA/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1 https://www.amazon.com/dp/B00555X8OA/ref=dp-kindle-redirect?...
- neuro_imager 9y ago
  Overall, I like the idea of a metric such as the one you describe. However, a change in one year mortality rate is unlikely to be a reliable indicator of the efficiency of most treatments. Most medical interventions have more modest effects than definite improvement in short term survival.
  - MarkMc 9y ago
    Sure, there might be better metrics than 1-year survival and different treatments could have different metrics.

nradov 9y ago

Nice theory, but in reality there's no way to do accurate risk adjustment. We just don't have enough data and so many physicians would end up unfairly penalized due to factors outside their control. Perhaps in 100 years we'll be in a better position to do something like that.

tinokid 9y ago
Is this an argument against tracking and releasing data? Let people do their own risk adjustments. If twice as many people who are going to one doctor or hospital are ending up dead or crippled, and there are no discernible confounding factors (age, socioeconomic, co-morbidities, etc), then I want to know. One might even say the patient has a right to know. The burden should be on the provider/facility to convince the patient to trust them nonetheless.
- nradov 9y ago
  Yes this is an argument against releasing data. You and most other patients lack the skills and context to interpret the data in a meaningful way. And no provider currently captures all of the confounding factors. In fact, for most serious conditions we haven't even done enough research to understand what all the confounding factors are. If we go with your proposal then the inevitable outcome is that the best providers (particularly surgeons) will engage in metrics arbitrage by refusing to treat patients whose co-morbidities and complications aren't adequately captured by standardized coding systems and clinical guidelines. Is that really the outcome you want? As for putting the burden on providers to convince patients to trust them, good luck with that. We currently have shortages of providers in many areas and specialties due to price fixing and supply constraints. So most patients have to take what they can get regardless of trust.
  - tinokid 9y ago
    Your views on bodily autonomy seem bizarre. Patients own their own bodies and have the right to make fully informed decisions about what happens to them. If a provider's or facility's (or entire speciality's) numbers look bad, that's important information for the consumer to possess. Hiding the numbers should not be an option. Or do you think that car crash test results, and airplane crash data, and health department inspection findings should also be kept secret? More problematically, if you can't articulate a quantifiable standard that will indicate a provider's quality level, then then entire concept seems ill-defined. I.e., when you refer to the "best providers", what does that even mean?

adrianN 9y ago

The problem is not randomly adding a couple of minutes that you can average out again, but systemic biases like the one your parent mentioned: difficult cases are treated by different doctors than easier cases.