4 ms·

I could see that problem occurring if the metric was “what is the success rate of everything that Surgeon X does”. I can’t see that problem occurring for “Surge

I could see that problem occurring if the metric was “what is the success rate of everything that Surgeon X does”. I can’t see that problem occurring for “Surgeon X performing Procedure Y has N% of patients reporting relief and M% of patients reporting complications after the surgery”. What am I missing here? Edit: Follow-up question: notwithstanding the dysfunction of congress and the ability of companies to find loopholes, and assuming no universal health care to eliminate the role of insurers, surely a solution would be to prohibit the use of this information in the same way that ACA prohibits the use of pre-existing conditions to deny coverage?

aerostable_slug 5y ago
Surgeons would steer away from difficult, lower-percentage-outcome procedures which is precisely what you don't want. You want risk-takers who repeatedly tackle the surgeries and (ideally) get a more positive outcome percentage than a newbie.
- riku_iki 5y ago
  Meaning they will not take over my risky surgery, and I will still be choosing among risk takers, but with clear stats in my hands.
rscho 5y ago
Surgeons will concentrate on easy procedures and will basically all have an almost identical track record of quasi perfection. So there will be nobody left to perform procedure Y, where Y has an intrinsically high rate of failure. Same thing for difficult patients. No one will touch Mr X who's got a complex problem.
lcuff 5y ago
In my mind it gets worse when the procedure is identified: A difficult shoulder surgery by an orthopedic surgeon gets refused because it might be result in a mediocre outcome and lower his 'success' rate. The patient can't get the surgery because no surgeon wants to 'risk' his success rate numbers.
- grog454 5y ago
  Seems like a good system to me. If a doctor expects a lower than average success rate for performing some specific operation, he should let some other doctor do it anyway.
  - rscho 5y ago
    Other doctors? And where will you find those? No one will ever touch you, except if you pay more. Pay more for worse outcomes, really.
  - lcuff 5y ago
    In my mind it is not a good system, because if the surgeon is looking at a much-more-difficult than average circumstance, than he could reasonably expect a lower-than-average result, but a result that might be much better than average for a case so difficult. As a separate observation, any time data is kept, turned into metrics that then become the basis for goals ("I want to have a better-than-average success rate, as a surgeon") then the system gets gamed. I had a boss once propose to down-rate agile teams that didn't get done everything they took on in a sprint. He apparently didn't realize that teams would immediate game the system by taking on less actual work. They could up their 'point' estimates for each task, and always get the work done.
    - grog454 5y ago
      I think you are right that there is a natural and understandable psychological resistance to a data-based evaluation system. I understand that the doctor may realize something about the patient that will lower his/her chances of success. I'm arguing that a sufficiently comprehensive system would take in to account whatever that doctor realized (and perhaps much more) and compensate for it when determining expected outcomes.
      - lcuff 5y ago
        I'm guessing that the bureaucracy of such a system would be a significant burden. To avoid having the system badly gamed, you'd need a second evaluation, yes? By a _neutral_ party, (not a doctor working in the same practice, e.g.) At added cost and time. Not to mention I've heard more than one surgeon say "That was more difficult than I expected based on the imaging ... once we got in there we found <>". I'm also arguing that it's not just a psychological resistance to a data-based evaluation system. That people understand the system would be subject to being gamed, and the overall quality of the work would actually suffer. (A bit analogous to how peer-review and the tenure game has interfered with good science practices.)
sxg 5y ago
I'm guessing you haven't heard of Goodhart's Law? (https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law) Under your proposal, surgeons will be incentivized to selectively operate on easier patients and minimize their complication rates while not performing surgery on very sick patients who may also need the same surgery. Different surgeons in different areas treat different kinds of patients. It's hard to accurately measure anything in a meaningful way that should influence decision making. To use your example, surgeon Z may also perform procedure Y but has (N-5)% of patients reporting relief and (M+5)% of patients reporting post-op complications. However, surgeon Z works at a community hospital and treats a poorer patient population with more co-morbidities. Can you really say if surgeon X is better than surgeon Z?
- grog454 5y ago
  Is it really too "hard" to perform rigorous statical analysis? Why can't you factor in patient genetics + background + circumstances to come up with some expected chance of success for each procedure (in fact, isn't that why doctors have such detailed patient histories?). Isn't it the doctor's job to estimate and inform the patient of the expected outcomes? Doctor's historical success rate exceeds expected success rate on average => Good (or lucky) doctor.
  - rscho 5y ago
    The number of factors you'd have to consider to achieve that is huge enough that it makes it completely unrealistic, both practically and financially speaking. In fact, I'm 100% convinced that it's currently impossible to build such a system, given the extent of medical knowledge.
    - grog454 5y ago
      I'm skeptical. I concede it's probably a hard problem but there is an entire field dedicated to hard statistical problems called data science. What is the point of having detailed patient histories and data if it can't be used to inform decisions?
tlb 5y ago
Doctors that work on older patients or worse cases will have worse numbers. Usually that effect far outweighs the skill difference between doctors.
pishpash 5y ago
Is the aggregate number reported, i.e. aggregated across all surgeons for procedure Y?

aerostable_slug 5y ago

Surgeons would steer away from difficult, lower-percentage-outcome procedures which is precisely what you don't want. You want risk-takers who repeatedly tackle the surgeries and (ideally) get a more positive outcome percentage than a newbie.

riku_iki 5y ago
Meaning they will not take over my risky surgery, and I will still be choosing among risk takers, but with clear stats in my hands.

rscho 5y ago

Surgeons will concentrate on easy procedures and will basically all have an almost identical track record of quasi perfection. So there will be nobody left to perform procedure Y, where Y has an intrinsically high rate of failure. Same thing for difficult patients. No one will touch Mr X who's got a complex problem.

lcuff 5y ago

In my mind it gets worse when the procedure is identified: A difficult shoulder surgery by an orthopedic surgeon gets refused because it might be result in a mediocre outcome and lower his 'success' rate. The patient can't get the surgery because no surgeon wants to 'risk' his success rate numbers.

grog454 5y ago
Seems like a good system to me. If a doctor expects a lower than average success rate for performing some specific operation, he should let some other doctor do it anyway.
- rscho 5y ago
  Other doctors? And where will you find those? No one will ever touch you, except if you pay more. Pay more for worse outcomes, really.
- lcuff 5y ago
  In my mind it is not a good system, because if the surgeon is looking at a much-more-difficult than average circumstance, than he could reasonably expect a lower-than-average result, but a result that might be much better than average for a case so difficult. As a separate observation, any time data is kept, turned into metrics that then become the basis for goals ("I want to have a better-than-average success rate, as a surgeon") then the system gets gamed. I had a boss once propose to down-rate agile teams that didn't get done everything they took on in a sprint. He apparently didn't realize that teams would immediate game the system by taking on less actual work. They could up their 'point' estimates for each task, and always get the work done.
  - grog454 5y ago
    I think you are right that there is a natural and understandable psychological resistance to a data-based evaluation system. I understand that the doctor may realize something about the patient that will lower his/her chances of success. I'm arguing that a sufficiently comprehensive system would take in to account whatever that doctor realized (and perhaps much more) and compensate for it when determining expected outcomes.
    - lcuff 5y ago
      I'm guessing that the bureaucracy of such a system would be a significant burden. To avoid having the system badly gamed, you'd need a second evaluation, yes? By a _neutral_ party, (not a doctor working in the same practice, e.g.) At added cost and time. Not to mention I've heard more than one surgeon say "That was more difficult than I expected based on the imaging ... once we got in there we found <>". I'm also arguing that it's not just a psychological resistance to a data-based evaluation system. That people understand the system would be subject to being gamed, and the overall quality of the work would actually suffer. (A bit analogous to how peer-review and the tenure game has interfered with good science practices.)

sxg 5y ago

I'm guessing you haven't heard of Goodhart's Law? (https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law) Under your proposal, surgeons will be incentivized to selectively operate on easier patients and minimize their complication rates while not performing surgery on very sick patients who may also need the same surgery. Different surgeons in different areas treat different kinds of patients. It's hard to accurately measure anything in a meaningful way that should influence decision making. To use your example, surgeon Z may also perform procedure Y but has (N-5)% of patients reporting relief and (M+5)% of patients reporting post-op complications. However, surgeon Z works at a community hospital and treats a poorer patient population with more co-morbidities. Can you really say if surgeon X is better than surgeon Z?

grog454 5y ago
Is it really too "hard" to perform rigorous statical analysis? Why can't you factor in patient genetics + background + circumstances to come up with some expected chance of success for each procedure (in fact, isn't that why doctors have such detailed patient histories?). Isn't it the doctor's job to estimate and inform the patient of the expected outcomes? Doctor's historical success rate exceeds expected success rate on average => Good (or lucky) doctor.
- rscho 5y ago
  The number of factors you'd have to consider to achieve that is huge enough that it makes it completely unrealistic, both practically and financially speaking. In fact, I'm 100% convinced that it's currently impossible to build such a system, given the extent of medical knowledge.
  - grog454 5y ago
    I'm skeptical. I concede it's probably a hard problem but there is an entire field dedicated to hard statistical problems called data science. What is the point of having detailed patient histories and data if it can't be used to inform decisions?

tlb 5y ago

Doctors that work on older patients or worse cases will have worse numbers. Usually that effect far outweighs the skill difference between doctors.

pishpash 5y ago

Is the aggregate number reported, i.e. aggregated across all surgeons for procedure Y?