5 ms·

This is so frustrating for me personally - I recently had to choose between several hospitals without knowing anything about the historical patient outcomes for

by MarkMc 10y ago

This is so frustrating for me personally - I recently had to choose between several hospitals without knowing anything about the historical patient outcomes for each hospital. A hundred years ago a doctor named Ernest Codman suggested that hospitals should be rated by an "End Result System" [1] where paient outcomes were measured and compared, yet only now do we have the "first comprehensive study comparing how well individual hospitals treated a variety of medical conditions". And we still do not know exactly which hospital is best and which is worst. To me the best way to judge a hospital is blindingly obvious: For each patient at admission, estimate the chance he/she will be alive after 3 years. For example, a 65-year-old woman who is a smoker, is overweight and has stage 2 lung cancer might have a 25% chance of being alive in 3 years. Then compare the predicted outcome with the actual outcome. For example, say both Hospital A and Hospital B both admit 100 patients with 20% predicted chance of surviving 3 years. If after 3 years there are 30 such patients from Hospital A still alive but only 10 such patients from Hospital B, it would strongly indicate that Hospital A is better. We could look then at Hospital A for ways to improve Hospital B. What's so disheartening is that this process doesn't involve any special technology. It could have been implemented 100 years ago, albeit with less rigorous statistical methods. We spend a billion dollars on evaluating a drug which has a marginal, one-off benfit for a few hundred thousand people. But we ignore a process which is cheap and offers long-lasting benefits to millions of people. It seems that the medical establishment is too powerful in this case. [1] https://en.wikipedia.org/wiki/Ernest_Amory_Codman https://en.wikipedia.org/wiki/Ernest_Amory_Codman

nradov 10y ago
It's not that simple, and it only seems blindingly obvious to you because you haven't actually worked in healthcare. The trouble with using survival or outcome ratings is that they effectively punish providers who take on the hardest, riskiest cases. We simply don't have enough accurate data. And many of the input variables aren't independent. The reality is that's it's impossible to accurately quantify all the different factors which potentially impact survival. So some providers find ways to game the system through metrics arbitrage. That said, it's still worthwhile to gather and use it for quality improvement. We just have to be careful to treat it with appropriate caveats and look at qualitative factors as well as quantified metrics.
- monkmartinez 10y ago
  Spot on analysis. The complexity of the human factor, for want of a better term, in healthcare is so hard to measure. I can only relate it to what I know. That is, I work with several paramedics that are just "good" at differential diagnosis. They would probably be excellent doctors, but life didn't deal those cards in. I also work with "meh" paramedics that, if given the choice, I wouldn't want taking care of my family. They do an adequate job to the bare minimum standard set, but they are not great by any stretch of the imagination. Both paramedics charge the same and carry the same title. Both could miss something critical, or not. The fact is they are both humans practicing paramedicine. Until we automate the variability out of healthcare, we are stuck with inconsistent results. Consider that roughly 70 years ago, lobotomies were considered a valid medical procedure for psychiatric problems including anxiety and depression.
  - fapjacks 10y ago
    Maybe you will be surprised to learn that lobotomies are considered valid medical procedures to this day.
- MarkMc 10y ago
  Sorry yes, I accept it is more complex than I described. There is the possibility that (a) the hospital admitting a patient can assess patient risk more accurately than a university professor who specialises in risk factors; (b) the size of the hospital advantage is significant when compared to the real differences in risk between hospitals; and (c) the hospital takes advantage of its better analysis by turning away patients where the professor risk rating of the patient is higher than the hospital risk rating. This is subtly different to your analysis in two ways: 1. The trouble with using survival or outcome ratings is that they effectively punish providers who take on the hardest, riskiest cases. Not necessarily - if the professor underestimates the risk in easy cases and overestimates the risk in hard cases it would give the hospital an incentive to take on the hard cases and turn away the easy cases. 2. The reality is that's it's impossible to accurately quantify all the different factors which potentially impact survival. In some cases we have remarkably detailed understanding of a broad range of risk factors - take a look at table 3 from page 23 in this [1] report on stem cell transplants. There are detailed odds ratios for patient age, ethnicity, karnofsky score, CMV status, disease type and stage, interval between diagnosis and transplant, year of transplant, donor match type, donor/recipient sex parity and donor age. More importantly, the professor doesn't need to be particularly accurate in assessing patient risk - she just needs to be as accurate as the hospital. (Or close enough that the difference would not give the hospital a significant advantage) Can the professor get close enough to hospital accuracy when estimating patient risk? Two relevant points spring to mind: (a) In the book Thinking Fast and Slow there is a description of how expert analysis can often be matched by simple rules of thumb. (b) The book Superforcasting describes how some lay people are able to make better predictions than the CIA about future geopolitical events, despite having no access to classified information. My gut feel is that the hospital would not have a significant edge over the professor. But I'm willing to be proved wrong - let's conduct an experiment where a doctor estimates the 3-year chance of survival for each patient at the time the patient is admitted to hospital, and a professor also makes the same estimate based only on the patient's database record. The winner gets half a million dollars. Wouldn't this experiment give us valuable insight in determining whether hospitals can game an End Results system? [1] http://www.seattlecca.org/sites/default/files/content_page/2016-07/9f/inline-documents/CIBMTR-Final-2015-Transplant-Center-Specific-Survival-Report.pdf http://www.seattlecca.org/sites/default/files/content_page/2...
bigchewy 10y ago
any time a problem of this magnitude hasn't been solved AND you think the answer is 'blindingly obvious', consider that you don't fully grok the problem. I've been in healthcare analytics for 10+ years and it's incredibly difficult to measure outcomes. Take your example of 3 year outcomes, are you saying that the entire reason why someone is alive is because of the hospital they went to 3 years ago? There are so many confounding variables that even trying to measure outcomes 90 days out is difficult. On top of that, the other commenter mentioned the unintended consequences of simplistic measurements like the one you describe. Look at measuring patient satisfaction - 'what could possibly be wrong with measuring patient satisfaction', people say! hold docs/hospitals/etc. accountable. but when you look at the data, patient satisfaction scores are weakly correlated with evidence based care and healthy outcomes but it is strongly correlated with things like how much time the patient sat in the waiting room or if the doctor prescribed them a medication.
- Broken_Hippo 10y ago
  > There are so many confounding variables that even trying to measure outcomes 90 days out is difficult. Very much this. Two patients, same doctor, same proceedure, with somewhat similar medical backgrounds. Nearly the same progression in the hospital, and released to go home, and possibly to doctors that don't interact with that hospital. Perhaps one is poor and sometimes skips medication. Only one starts eating healthily and starts exercise. One has a stressful home life. Maybe one, when the recovery isn't going as quickly as possible, starts mixing in alternative medicines but doesn't tell his GP. Maybe they were both women, in the hospital to give birth - and one got pregnant again in the first year while the other didn't. These sorts of things greatly affect the outcomes, but none of them are under the hospital's controls.
- MarkMc 10y ago
  > any time a problem of this magnitude hasn't been solved AND you think the answer is 'blindingly obvious', consider that you don't fully grok the problem. That is a good point - this thread has turned out to be more complex than I expected, and had I been a bit more open-minded I would have avoided the phrase 'blindingly obvious'. Still, I'm not yet convinced this is an intractable problem. The New York Times articles says that patients are 3 times more likely to die at one hospital than another, even after controlling for patient sickness, income and age. This may be due to (a) a confounding variable; (b) a real difference between the quality of care provided by the hospitals. How much weight should we place on each possibility? Is it 50/50, or 60/40, or 10/90? If you have to have a medical procedure and were told the survival rate of Hospital A is 98% and at Hospital B is 94% would you say, "there are so many possible confounding variables, this statistic will have no bearing on my choice of hospital"?
deleted 10y ago
[deleted]

by MarkMc 10y ago

nradov 10y ago

It's not that simple, and it only seems blindingly obvious to you because you haven't actually worked in healthcare. The trouble with using survival or outcome ratings is that they effectively punish providers who take on the hardest, riskiest cases. We simply don't have enough accurate data. And many of the input variables aren't independent. The reality is that's it's impossible to accurately quantify all the different factors which potentially impact survival. So some providers find ways to game the system through metrics arbitrage. That said, it's still worthwhile to gather and use it for quality improvement. We just have to be careful to treat it with appropriate caveats and look at qualitative factors as well as quantified metrics.

monkmartinez 10y ago
Spot on analysis. The complexity of the human factor, for want of a better term, in healthcare is so hard to measure. I can only relate it to what I know. That is, I work with several paramedics that are just "good" at differential diagnosis. They would probably be excellent doctors, but life didn't deal those cards in. I also work with "meh" paramedics that, if given the choice, I wouldn't want taking care of my family. They do an adequate job to the bare minimum standard set, but they are not great by any stretch of the imagination. Both paramedics charge the same and carry the same title. Both could miss something critical, or not. The fact is they are both humans practicing paramedicine. Until we automate the variability out of healthcare, we are stuck with inconsistent results. Consider that roughly 70 years ago, lobotomies were considered a valid medical procedure for psychiatric problems including anxiety and depression.
- fapjacks 10y ago
  Maybe you will be surprised to learn that lobotomies are considered valid medical procedures to this day.
MarkMc 10y ago
Sorry yes, I accept it is more complex than I described. There is the possibility that (a) the hospital admitting a patient can assess patient risk more accurately than a university professor who specialises in risk factors; (b) the size of the hospital advantage is significant when compared to the real differences in risk between hospitals; and (c) the hospital takes advantage of its better analysis by turning away patients where the professor risk rating of the patient is higher than the hospital risk rating. This is subtly different to your analysis in two ways: 1. The trouble with using survival or outcome ratings is that they effectively punish providers who take on the hardest, riskiest cases. Not necessarily - if the professor underestimates the risk in easy cases and overestimates the risk in hard cases it would give the hospital an incentive to take on the hard cases and turn away the easy cases. 2. The reality is that's it's impossible to accurately quantify all the different factors which potentially impact survival. In some cases we have remarkably detailed understanding of a broad range of risk factors - take a look at table 3 from page 23 in this [1] report on stem cell transplants. There are detailed odds ratios for patient age, ethnicity, karnofsky score, CMV status, disease type and stage, interval between diagnosis and transplant, year of transplant, donor match type, donor/recipient sex parity and donor age. More importantly, the professor doesn't need to be particularly accurate in assessing patient risk - she just needs to be as accurate as the hospital. (Or close enough that the difference would not give the hospital a significant advantage) Can the professor get close enough to hospital accuracy when estimating patient risk? Two relevant points spring to mind: (a) In the book Thinking Fast and Slow there is a description of how expert analysis can often be matched by simple rules of thumb. (b) The book Superforcasting describes how some lay people are able to make better predictions than the CIA about future geopolitical events, despite having no access to classified information. My gut feel is that the hospital would not have a significant edge over the professor. But I'm willing to be proved wrong - let's conduct an experiment where a doctor estimates the 3-year chance of survival for each patient at the time the patient is admitted to hospital, and a professor also makes the same estimate based only on the patient's database record. The winner gets half a million dollars. Wouldn't this experiment give us valuable insight in determining whether hospitals can game an End Results system? [1] http://www.seattlecca.org/sites/default/files/content_page/2016-07/9f/inline-documents/CIBMTR-Final-2015-Transplant-Center-Specific-Survival-Report.pdf http://www.seattlecca.org/sites/default/files/content_page/2...

bigchewy 10y ago

any time a problem of this magnitude hasn't been solved AND you think the answer is 'blindingly obvious', consider that you don't fully grok the problem. I've been in healthcare analytics for 10+ years and it's incredibly difficult to measure outcomes. Take your example of 3 year outcomes, are you saying that the entire reason why someone is alive is because of the hospital they went to 3 years ago? There are so many confounding variables that even trying to measure outcomes 90 days out is difficult. On top of that, the other commenter mentioned the unintended consequences of simplistic measurements like the one you describe. Look at measuring patient satisfaction - 'what could possibly be wrong with measuring patient satisfaction', people say! hold docs/hospitals/etc. accountable. but when you look at the data, patient satisfaction scores are weakly correlated with evidence based care and healthy outcomes but it is strongly correlated with things like how much time the patient sat in the waiting room or if the doctor prescribed them a medication.

Broken_Hippo 10y ago
> There are so many confounding variables that even trying to measure outcomes 90 days out is difficult. Very much this. Two patients, same doctor, same proceedure, with somewhat similar medical backgrounds. Nearly the same progression in the hospital, and released to go home, and possibly to doctors that don't interact with that hospital. Perhaps one is poor and sometimes skips medication. Only one starts eating healthily and starts exercise. One has a stressful home life. Maybe one, when the recovery isn't going as quickly as possible, starts mixing in alternative medicines but doesn't tell his GP. Maybe they were both women, in the hospital to give birth - and one got pregnant again in the first year while the other didn't. These sorts of things greatly affect the outcomes, but none of them are under the hospital's controls.
MarkMc 10y ago
> any time a problem of this magnitude hasn't been solved AND you think the answer is 'blindingly obvious', consider that you don't fully grok the problem. That is a good point - this thread has turned out to be more complex than I expected, and had I been a bit more open-minded I would have avoided the phrase 'blindingly obvious'. Still, I'm not yet convinced this is an intractable problem. The New York Times articles says that patients are 3 times more likely to die at one hospital than another, even after controlling for patient sickness, income and age. This may be due to (a) a confounding variable; (b) a real difference between the quality of care provided by the hospitals. How much weight should we place on each possibility? Is it 50/50, or 60/40, or 10/90? If you have to have a medical procedure and were told the survival rate of Hospital A is 98% and at Hospital B is 94% would you say, "there are so many possible confounding variables, this statistic will have no bearing on my choice of hospital"?

deleted 10y ago

[deleted]