4 ms·

Like anything in life the devil is in the details. What measures and metrics could possibly be used to determine the score? If you go by mortality rates alone

Like anything in life the devil is in the details. What measures and metrics could possibly be used to determine the score? If you go by mortality rates alone you risk creating an environment where no one wants to operate on the most at risk patients. You'll quickly find that the best surgeons aren't necessarily the most surgically skilled, but instead those who do the best job of pre-screening surgical candidates. If this continues, soon you'll have patients whom doctors flat out refuse to cut open. If instead of mortality rates, you score the doctors on skill of movement in their procedures, again, you'll find doctors gaming the system by picking healthier patients with lower body fat percentages. Fundamentally, if a system was in place to score surgeons a lot of checks and balances would need to be enacted to avoid lowering the quality of care by ensuring doctors, hospital staff, and administrators couldn't simply pick and choose what surgeon gets what patient. I really see this as a neat data science project after the fact, but if implemented could have a significant downside impact on patient outcomes for some.

Kliment 12y ago
Well, the solution proposed in the article is to keep the scores hidden from everyone but the particular surgeon. That is, have the score be a motivator and guide for personal improvement rather than an external quality indicator. They report on trials where similar scores were published in NY and the end result was that the highest risk patients got shipped off to Ohio, exactly as you describe. So it's critical that the data is not shown to anyone but the surgeon in question.
- patcheudor 12y ago
  The problem is that by keeping it hidden from everyone but the surgeon and by still allowing the surgeon to have a say in who they operate on, they can still game the system. There are no external controls at that point. Now if it's only for their benefit what possible reason would a surgeon have to game the score you might ask? You can provide all the assurances in the world that the score is a personal motivator, but we all know that could change at the drop of the hat. I've seen this far too often in the security field. Someone will come up with a great measure, will reassure everyone that all unintended consequences have been considered and then boom. A year down the line an executive, unaware of the ramifications of unintended consequences will want to use the metric for performance evaluations. I think any surgeon who believes the score is only there to help them and that they shouldn't plan for a future where someone pushes to make it more public is being a bit naive in the ways of the world. This isn't to say I don't think there could be a solution. The scoring system needs to be built from the ground up with an offset risk equation which provides significant incentive to operate on at risk patients. Maybe a surgeon gets three points for operating on the patient and if they die they only loose one. However if they have a healthy patient maybe they start off with one point, with just one to loose. Obviously all the unintended consequences for this model would need to be explored at length.
  - jerf 12y ago
    "There are no external controls at that point." Well... there appear to be no external controls at this point, either...
    - baddox 12y ago
      Right. Even in the current situation, surely there is some incentive for surgeons to turn down high-risk patients, because of the inevitable stress and potential legal trouble from a failed operation.
- wesleyy 12y ago
  If the score exists, the top end surgeons are going to show it to their patients, which will force the low end surgeons to do so as well. The only way to keep the score secret is to design regulations for it which I just don't see happening.
visarga 12y ago
Surgeons would optimize for the test criteria instead of focusing on the operation.
k__ 12y ago
I agree, but it could be multi-dimensional. How many survived and how critical were they. So the score would reflect if the surgeon avoided critical patients.

Kliment 12y ago

Well, the solution proposed in the article is to keep the scores hidden from everyone but the particular surgeon. That is, have the score be a motivator and guide for personal improvement rather than an external quality indicator. They report on trials where similar scores were published in NY and the end result was that the highest risk patients got shipped off to Ohio, exactly as you describe. So it's critical that the data is not shown to anyone but the surgeon in question.

patcheudor 12y ago
The problem is that by keeping it hidden from everyone but the surgeon and by still allowing the surgeon to have a say in who they operate on, they can still game the system. There are no external controls at that point. Now if it's only for their benefit what possible reason would a surgeon have to game the score you might ask? You can provide all the assurances in the world that the score is a personal motivator, but we all know that could change at the drop of the hat. I've seen this far too often in the security field. Someone will come up with a great measure, will reassure everyone that all unintended consequences have been considered and then boom. A year down the line an executive, unaware of the ramifications of unintended consequences will want to use the metric for performance evaluations. I think any surgeon who believes the score is only there to help them and that they shouldn't plan for a future where someone pushes to make it more public is being a bit naive in the ways of the world. This isn't to say I don't think there could be a solution. The scoring system needs to be built from the ground up with an offset risk equation which provides significant incentive to operate on at risk patients. Maybe a surgeon gets three points for operating on the patient and if they die they only loose one. However if they have a healthy patient maybe they start off with one point, with just one to loose. Obviously all the unintended consequences for this model would need to be explored at length.
- jerf 12y ago
  "There are no external controls at that point." Well... there appear to be no external controls at this point, either...
  - baddox 12y ago
    Right. Even in the current situation, surely there is some incentive for surgeons to turn down high-risk patients, because of the inevitable stress and potential legal trouble from a failed operation.
wesleyy 12y ago
If the score exists, the top end surgeons are going to show it to their patients, which will force the low end surgeons to do so as well. The only way to keep the score secret is to design regulations for it which I just don't see happening.

visarga 12y ago

Surgeons would optimize for the test criteria instead of focusing on the operation.

k__ 12y ago

I agree, but it could be multi-dimensional. How many survived and how critical were they. So the score would reflect if the surgeon avoided critical patients.