9 ms·

> When I ask for hard data on the surgeon’s actual success rate for this type of surgery, well, they don’t track that—but look, just trust me, the guy’s reallly

by rscho 5y ago

> When I ask for hard data on the surgeon’s actual success rate for this type of surgery, well, they don’t track that—but look, just trust me, the guy’s realllyyy good. For good reason. Tracking of clinical outcomes is the wet dream of insurance companies. It's very toxic to the healthcare system, because it pushes practitioners to focus on easy cases where a good outcome is expected and causes major inequalities in access to healthcare. I'm not saying the present situation is ideal, but for the system as a whole in its current form, tracking clinical outcomes is a very bad idea.

csnover 5y ago
I could see that problem occurring if the metric was “what is the success rate of everything that Surgeon X does”. I can’t see that problem occurring for “Surgeon X performing Procedure Y has N% of patients reporting relief and M% of patients reporting complications after the surgery”. What am I missing here? Edit: Follow-up question: notwithstanding the dysfunction of congress and the ability of companies to find loopholes, and assuming no universal health care to eliminate the role of insurers, surely a solution would be to prohibit the use of this information in the same way that ACA prohibits the use of pre-existing conditions to deny coverage?
- aerostable_slug 5y ago
  Surgeons would steer away from difficult, lower-percentage-outcome procedures which is precisely what you don't want. You want risk-takers who repeatedly tackle the surgeries and (ideally) get a more positive outcome percentage than a newbie.
  - riku_iki 5y ago
    Meaning they will not take over my risky surgery, and I will still be choosing among risk takers, but with clear stats in my hands.
- rscho 5y ago
  Surgeons will concentrate on easy procedures and will basically all have an almost identical track record of quasi perfection. So there will be nobody left to perform procedure Y, where Y has an intrinsically high rate of failure. Same thing for difficult patients. No one will touch Mr X who's got a complex problem.
- lcuff 5y ago
  In my mind it gets worse when the procedure is identified: A difficult shoulder surgery by an orthopedic surgeon gets refused because it might be result in a mediocre outcome and lower his 'success' rate. The patient can't get the surgery because no surgeon wants to 'risk' his success rate numbers.
  - grog454 5y ago
    Seems like a good system to me. If a doctor expects a lower than average success rate for performing some specific operation, he should let some other doctor do it anyway.
    - rscho 5y ago
      Other doctors? And where will you find those? No one will ever touch you, except if you pay more. Pay more for worse outcomes, really.
    - lcuff 5y ago
      In my mind it is not a good system, because if the surgeon is looking at a much-more-difficult than average circumstance, than he could reasonably expect a lower-than-average result, but a result that might be much better than average for a case so difficult. As a separate observation, any time data is kept, turned into metrics that then become the basis for goals ("I want to have a better-than-average success rate, as a surgeon") then the system gets gamed. I had a boss once propose to down-rate agile teams that didn't get done everything they took on in a sprint. He apparently didn't realize that teams would immediate game the system by taking on less actual work. They could up their 'point' estimates for each task, and always get the work done.
      - grog454 5y ago
        I think you are right that there is a natural and understandable psychological resistance to a data-based evaluation system. I understand that the doctor may realize something about the patient that will lower his/her chances of success. I'm arguing that a sufficiently comprehensive system would take in to account whatever that doctor realized (and perhaps much more) and compensate for it when determining expected outcomes.
- sxg 5y ago
  I'm guessing you haven't heard of Goodhart's Law? (https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law) Under your proposal, surgeons will be incentivized to selectively operate on easier patients and minimize their complication rates while not performing surgery on very sick patients who may also need the same surgery. Different surgeons in different areas treat different kinds of patients. It's hard to accurately measure anything in a meaningful way that should influence decision making. To use your example, surgeon Z may also perform procedure Y but has (N-5)% of patients reporting relief and (M+5)% of patients reporting post-op complications. However, surgeon Z works at a community hospital and treats a poorer patient population with more co-morbidities. Can you really say if surgeon X is better than surgeon Z?
  - grog454 5y ago
    Is it really too "hard" to perform rigorous statical analysis? Why can't you factor in patient genetics + background + circumstances to come up with some expected chance of success for each procedure (in fact, isn't that why doctors have such detailed patient histories?). Isn't it the doctor's job to estimate and inform the patient of the expected outcomes? Doctor's historical success rate exceeds expected success rate on average => Good (or lucky) doctor.
    - rscho 5y ago
      The number of factors you'd have to consider to achieve that is huge enough that it makes it completely unrealistic, both practically and financially speaking. In fact, I'm 100% convinced that it's currently impossible to build such a system, given the extent of medical knowledge.
      - grog454 5y ago
        I'm skeptical. I concede it's probably a hard problem but there is an entire field dedicated to hard statistical problems called data science. What is the point of having detailed patient histories and data if it can't be used to inform decisions?
- tlb 5y ago
  Doctors that work on older patients or worse cases will have worse numbers. Usually that effect far outweighs the skill difference between doctors.
- pishpash 5y ago
  Is the aggregate number reported, i.e. aggregated across all surgeons for procedure Y?
commandar 5y ago
To some degree, there's an even more toxic element of this already in play with the amount of weight the wider US medical system puts behind patient satisfaction surveys. Many times, things that a patient wants and would make a patient happy are medically contraindicated and lead to worse outcomes, yet there's immense pressure on clinicians to maintain patient satisfaction metrics. I'm not disagreeing with you at all; more suggesting that we're currently relying on metrics that are even more perilous than actual clinical outcomes.
- Fomite 5y ago
  This is even true when we're not talking about patient satisfaction. There's definitely conflicts for things like infection control, where what we should probably should be measuring (process measures) conflict with what patient advocacy groups care about and are pushing for (deaths).
- henrikschroder 5y ago
  In patient satisfaction surveys, treatments like invasive surgery usually ranks very low, while treatments like massage rank very highly. One probably saved your life, the other just felt nice, and yet the second one is rated higher. It's complete insanity to even being comparing treatments that are so different.
  - lazide 5y ago
    Also getting antibiotics for something gets rated pretty high, even if it did worse than nothing.
wins32767 5y ago
Risk adjusted scoring is entirely viable in this age of data science, there is just no appetite for it. Doctors fight very hard against it because who actually wants to be held accountable for outcomes? Insurance companies really aren't the villains in the US healthcare system, they're going to make money no matter what because they pass cost increases on to their subscribers and are capped in how much profit they can make via regulation.
- rscho 5y ago
  Risk adjusted scoring is currently absolutely not viable. You underestimate the messiness of the healthcare system by a huge margin. We don't even manage to record basic vital signs consistently, so believe me when I tell you that you can forget about any kind of nice statistical trick given the weakness of our data gathering processes. Plus it's not a matter of objectivity. Surgeons will subjectively assess that doing easy cases will be better. And in addition, they'll be correct. That's what really matters.
  - wins32767 5y ago
    I literally was at a startup that could do this successfully, so I'm not underestimating the messiness.
    - rscho 5y ago
      I doubt we're speaking of the same thing, here.
- AussieWog93 5y ago
  >Risk adjusted scoring is entirely viable in this age of data science It's just not unviable, I don't think it's even possible. As soon as a metric becomes tracked, people are incentivized to game it. From my own experience as a very high-volume eBay seller, mandating a certain return rate led us to simply discourage customers from using the (convenient, well-designed) integrated returns systems. Mandating that only a tiny fraction of a percentage of items can be cancelled due to being out of stock leads to sellers sending either the wrong item or a fake tracking number (this gets us all the time on AliExpress). If data-driven software companies can't handle it for something as simple as eCommerce, I have no idea how the medical industry is supposed to get it right.
asmithmd1 5y ago
There are some things that are helpful that they will answer that have steered me away from a surgeon: How many of these surgeries do you do per year? An answer of 50 or above is good. What is YOUR rate of <specific complication mentioned in the consent>? Keep pushing until they tell you their rate, not the overall rate.
- rscho 5y ago
  They almost universally don't know their own rate. I don't know my own rates of complications. But, if I want you to go away because you seem to be the kind of guy that will come back to bite me, I'll gladly tell you I have a huge rate of complication.
  - asmithmd1 5y ago
    For a colonoscopy I asked about the risk of a perforated bowel. Their first response was, “that is meaningless, if it happens to you the rate is 100%.” I said if the chances are 50:50, then I am not doing it. They said it has happened twice in their career and based on my lack of risk factors, it would not happen to me. For my daughters tonsillectomy, the doctor was very happy to share how her stats for post surgery bleeding compared to both other doctors in her group, and the national average. But I live in a Boston suburb and every doctor is a lecturer at either Harvard or Mass General. Another question to ask is will an intern take part in the surgery. At teaching hospitals the answer is almost always yes. You can ask if they operate at any other hospitals, and again the answer is almost always yes, they operate at a suburban, non-teaching hospital where they will be the only one operating. I got a little bit humbled at Boston Childrens Hospital. I was doing some Googling about the risks of a CAT scan and asked if they did low-dose ones. They informed me that they in fact invented that procedure. Sure enough, the paper I was looking at was authored by a doctor on their staff.
    - rscho 5y ago
      I have 10+ years of clinical experience in academic hospitals, and have worked in Boston at Brigham and Women's. From this experience, I can tell you 2 things: 1. Being a lecturer at Harvard does not correlate with being a skillful clinician 2. Your view of the clinical system is very skewed, and will bring you more risks than benefits.
ropable 5y ago
On the other hand: tracking clinical outcomes actually tells us if a treatment actually works - a net positive for the species, I'd have thought.
- tlb 5y ago
  Doing RCTs of a treatment tells you if it works. Without random assignment (or some clever experimental design) tracking outcomes really doesn't tell you anything.
Schroedingersat 5y ago
> For good reason. Tracking of clinical outcomes is the wet dream of insurance companies. Sounds like a sufficient reason for single payer healthcare and single payer malpractise compensation even if all the imagined downsides were real.
pishpash 5y ago
You can forbid insurance from using that data by policy. It sounds like the same argument against measuring teacher performance because "toxic".
MichaelBurge 5y ago
Have one person do nothing but estimate the difficulty of a case(and be tracked and judged for the quality of their estimates), and the surgeon be tracked and judged on the relative performance given the difficulty. Then you could make a career out of doing well on difficult cases, or out of doing better-than-average on easy cases, and either would be viable.
refurb 5y ago
It's also a bad idea because the best surgeons often take on the most difficult cases you can't necessarily compare doctor to doctor without knowing the types of patients they treat.
billfruit 5y ago
Wouldn't that be rectifiable by also tracking the statistics of cases which a practitioner reneges or refuses. Then somebody with high success rate and high refusal rate stands out as a red flag.

by rscho 5y ago

csnover 5y ago

I could see that problem occurring if the metric was “what is the success rate of everything that Surgeon X does”. I can’t see that problem occurring for “Surgeon X performing Procedure Y has N% of patients reporting relief and M% of patients reporting complications after the surgery”. What am I missing here? Edit: Follow-up question: notwithstanding the dysfunction of congress and the ability of companies to find loopholes, and assuming no universal health care to eliminate the role of insurers, surely a solution would be to prohibit the use of this information in the same way that ACA prohibits the use of pre-existing conditions to deny coverage?

aerostable_slug 5y ago
Surgeons would steer away from difficult, lower-percentage-outcome procedures which is precisely what you don't want. You want risk-takers who repeatedly tackle the surgeries and (ideally) get a more positive outcome percentage than a newbie.
- riku_iki 5y ago
  Meaning they will not take over my risky surgery, and I will still be choosing among risk takers, but with clear stats in my hands.
rscho 5y ago
Surgeons will concentrate on easy procedures and will basically all have an almost identical track record of quasi perfection. So there will be nobody left to perform procedure Y, where Y has an intrinsically high rate of failure. Same thing for difficult patients. No one will touch Mr X who's got a complex problem.
lcuff 5y ago
In my mind it gets worse when the procedure is identified: A difficult shoulder surgery by an orthopedic surgeon gets refused because it might be result in a mediocre outcome and lower his 'success' rate. The patient can't get the surgery because no surgeon wants to 'risk' his success rate numbers.
- grog454 5y ago
  Seems like a good system to me. If a doctor expects a lower than average success rate for performing some specific operation, he should let some other doctor do it anyway.
  - rscho 5y ago
    Other doctors? And where will you find those? No one will ever touch you, except if you pay more. Pay more for worse outcomes, really.
  - lcuff 5y ago
    In my mind it is not a good system, because if the surgeon is looking at a much-more-difficult than average circumstance, than he could reasonably expect a lower-than-average result, but a result that might be much better than average for a case so difficult. As a separate observation, any time data is kept, turned into metrics that then become the basis for goals ("I want to have a better-than-average success rate, as a surgeon") then the system gets gamed. I had a boss once propose to down-rate agile teams that didn't get done everything they took on in a sprint. He apparently didn't realize that teams would immediate game the system by taking on less actual work. They could up their 'point' estimates for each task, and always get the work done.
    - grog454 5y ago
      I think you are right that there is a natural and understandable psychological resistance to a data-based evaluation system. I understand that the doctor may realize something about the patient that will lower his/her chances of success. I'm arguing that a sufficiently comprehensive system would take in to account whatever that doctor realized (and perhaps much more) and compensate for it when determining expected outcomes.
sxg 5y ago
I'm guessing you haven't heard of Goodhart's Law? (https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law) Under your proposal, surgeons will be incentivized to selectively operate on easier patients and minimize their complication rates while not performing surgery on very sick patients who may also need the same surgery. Different surgeons in different areas treat different kinds of patients. It's hard to accurately measure anything in a meaningful way that should influence decision making. To use your example, surgeon Z may also perform procedure Y but has (N-5)% of patients reporting relief and (M+5)% of patients reporting post-op complications. However, surgeon Z works at a community hospital and treats a poorer patient population with more co-morbidities. Can you really say if surgeon X is better than surgeon Z?
- grog454 5y ago
  Is it really too "hard" to perform rigorous statical analysis? Why can't you factor in patient genetics + background + circumstances to come up with some expected chance of success for each procedure (in fact, isn't that why doctors have such detailed patient histories?). Isn't it the doctor's job to estimate and inform the patient of the expected outcomes? Doctor's historical success rate exceeds expected success rate on average => Good (or lucky) doctor.
  - rscho 5y ago
    The number of factors you'd have to consider to achieve that is huge enough that it makes it completely unrealistic, both practically and financially speaking. In fact, I'm 100% convinced that it's currently impossible to build such a system, given the extent of medical knowledge.
    - grog454 5y ago
      I'm skeptical. I concede it's probably a hard problem but there is an entire field dedicated to hard statistical problems called data science. What is the point of having detailed patient histories and data if it can't be used to inform decisions?
tlb 5y ago
Doctors that work on older patients or worse cases will have worse numbers. Usually that effect far outweighs the skill difference between doctors.
pishpash 5y ago
Is the aggregate number reported, i.e. aggregated across all surgeons for procedure Y?

commandar 5y ago

To some degree, there's an even more toxic element of this already in play with the amount of weight the wider US medical system puts behind patient satisfaction surveys. Many times, things that a patient wants and would make a patient happy are medically contraindicated and lead to worse outcomes, yet there's immense pressure on clinicians to maintain patient satisfaction metrics. I'm not disagreeing with you at all; more suggesting that we're currently relying on metrics that are even more perilous than actual clinical outcomes.

Fomite 5y ago
This is even true when we're not talking about patient satisfaction. There's definitely conflicts for things like infection control, where what we should probably should be measuring (process measures) conflict with what patient advocacy groups care about and are pushing for (deaths).
henrikschroder 5y ago
In patient satisfaction surveys, treatments like invasive surgery usually ranks very low, while treatments like massage rank very highly. One probably saved your life, the other just felt nice, and yet the second one is rated higher. It's complete insanity to even being comparing treatments that are so different.
- lazide 5y ago
  Also getting antibiotics for something gets rated pretty high, even if it did worse than nothing.

wins32767 5y ago

Risk adjusted scoring is entirely viable in this age of data science, there is just no appetite for it. Doctors fight very hard against it because who actually wants to be held accountable for outcomes? Insurance companies really aren't the villains in the US healthcare system, they're going to make money no matter what because they pass cost increases on to their subscribers and are capped in how much profit they can make via regulation.

rscho 5y ago
Risk adjusted scoring is currently absolutely not viable. You underestimate the messiness of the healthcare system by a huge margin. We don't even manage to record basic vital signs consistently, so believe me when I tell you that you can forget about any kind of nice statistical trick given the weakness of our data gathering processes. Plus it's not a matter of objectivity. Surgeons will subjectively assess that doing easy cases will be better. And in addition, they'll be correct. That's what really matters.
- wins32767 5y ago
  I literally was at a startup that could do this successfully, so I'm not underestimating the messiness.
  - rscho 5y ago
    I doubt we're speaking of the same thing, here.
AussieWog93 5y ago
>Risk adjusted scoring is entirely viable in this age of data science It's just not unviable, I don't think it's even possible. As soon as a metric becomes tracked, people are incentivized to game it. From my own experience as a very high-volume eBay seller, mandating a certain return rate led us to simply discourage customers from using the (convenient, well-designed) integrated returns systems. Mandating that only a tiny fraction of a percentage of items can be cancelled due to being out of stock leads to sellers sending either the wrong item or a fake tracking number (this gets us all the time on AliExpress). If data-driven software companies can't handle it for something as simple as eCommerce, I have no idea how the medical industry is supposed to get it right.

asmithmd1 5y ago

There are some things that are helpful that they will answer that have steered me away from a surgeon: How many of these surgeries do you do per year? An answer of 50 or above is good. What is YOUR rate of <specific complication mentioned in the consent>? Keep pushing until they tell you their rate, not the overall rate.

rscho 5y ago
They almost universally don't know their own rate. I don't know my own rates of complications. But, if I want you to go away because you seem to be the kind of guy that will come back to bite me, I'll gladly tell you I have a huge rate of complication.
- asmithmd1 5y ago
  For a colonoscopy I asked about the risk of a perforated bowel. Their first response was, “that is meaningless, if it happens to you the rate is 100%.” I said if the chances are 50:50, then I am not doing it. They said it has happened twice in their career and based on my lack of risk factors, it would not happen to me. For my daughters tonsillectomy, the doctor was very happy to share how her stats for post surgery bleeding compared to both other doctors in her group, and the national average. But I live in a Boston suburb and every doctor is a lecturer at either Harvard or Mass General. Another question to ask is will an intern take part in the surgery. At teaching hospitals the answer is almost always yes. You can ask if they operate at any other hospitals, and again the answer is almost always yes, they operate at a suburban, non-teaching hospital where they will be the only one operating. I got a little bit humbled at Boston Childrens Hospital. I was doing some Googling about the risks of a CAT scan and asked if they did low-dose ones. They informed me that they in fact invented that procedure. Sure enough, the paper I was looking at was authored by a doctor on their staff.
  - rscho 5y ago
    I have 10+ years of clinical experience in academic hospitals, and have worked in Boston at Brigham and Women's. From this experience, I can tell you 2 things: 1. Being a lecturer at Harvard does not correlate with being a skillful clinician 2. Your view of the clinical system is very skewed, and will bring you more risks than benefits.

ropable 5y ago

On the other hand: tracking clinical outcomes actually tells us if a treatment actually works - a net positive for the species, I'd have thought.

tlb 5y ago
Doing RCTs of a treatment tells you if it works. Without random assignment (or some clever experimental design) tracking outcomes really doesn't tell you anything.

Schroedingersat 5y ago

> For good reason. Tracking of clinical outcomes is the wet dream of insurance companies. Sounds like a sufficient reason for single payer healthcare and single payer malpractise compensation even if all the imagined downsides were real.

pishpash 5y ago

You can forbid insurance from using that data by policy. It sounds like the same argument against measuring teacher performance because "toxic".

MichaelBurge 5y ago

Have one person do nothing but estimate the difficulty of a case(and be tracked and judged for the quality of their estimates), and the surgeon be tracked and judged on the relative performance given the difficulty. Then you could make a career out of doing well on difficult cases, or out of doing better-than-average on easy cases, and either would be viable.

refurb 5y ago

It's also a bad idea because the best surgeons often take on the most difficult cases you can't necessarily compare doctor to doctor without knowing the types of patients they treat.

billfruit 5y ago

Wouldn't that be rectifiable by also tracking the statistics of cases which a practitioner reneges or refuses. Then somebody with high success rate and high refusal rate stands out as a red flag.