5 ms·

I'm not so sure Goodhart's Law is correct. For example, if I target a high win percentage, is win percentage, then, not a good measure? Mortality rates are a

I'm not so sure Goodhart's Law is correct. For example, if I target a high win percentage, is win percentage, then, not a good measure? Mortality rates are a pretty good measure for surgeons. The problem here is that they are inflating their measure by cherry picking. edit: Some measures should be targeted, the problem here is that it is done in an non genuine way

Buttons840 8y ago
Pro tip for those targeting a high win percentage. If you win your first game (beginners luck?), then stop playing and never play again. That's what the article is describing, surgeons won't even play the game, they won't even attempt to help certain people because it might hurt their "win percentage".
- natoliniak 8y ago
  another pro tip: compete only against weaker opponents (this is common in Boxing).
washadjeffmad 8y ago
Wins are likely from systems with fixed rules that don't reward innovation, so winning by the rules of a system doesn't necessarily mean a positive outcome. An example might be the system that lead to the housing crisis. Certainly a few people who gamed things the right way "won" big, but only at the expense of millions of homeowners. Mortality rates are one measure of many that can tell a story, but they're like incarceration rates in that they've become a perverse incentive that exacerbates an issue. If we say that putting more people in prison is a good thing, that's predicated on an assumption that only those that are being put in prisons are people worthy of being there, whose freedom is more costly to society than their imprisonment. When a district is rewarded for putting people in prisons, what types of things might you expect to happen to the legal systems and populations in those places? Would prosecutors then be incentivized to trump up charges and force people into serving prison time unnecessarily to make themselves look like bigger "winners"? Might police make more frivolous arrests or even stir up trouble in communities to paint a picture of rampant criminality so they can look to be "tough on crime"? Wouldn't you expect to see an all-causes decrease in long term crime? The problem isn't just cherry picking, but if people can game a system built around limited and gameable measures, then it's going to encourage min-maxing for profit.
LoSboccacc 8y ago
the problem here is that mortality rate is not correlated/normalized with the patient risk factor. the problem lies in the performance indicator itself, not strictly people trying to optimizing for what they are asked for. it is likely inhumane to refuse patients that have slightly above average risk - even if there are some considerations to be had, like having replacement organs go to waste, but that is already covered by the donor lists queues - but given people can find themselves out of work for having bad mortality rates, optimizing for the performance indicator it's the only logical conclusion. also, normalizing for the patient risk will likely just lead to people overstating patient risk, because once you reduce performance to an index, people will focus on what's measured before on what the intended goal for the measurement is.
- emodendroket 8y ago
  I'm not an expert in cardiatric medicine or anything, but couldn't patient risk be determined by objective factors like patient age and history of previous heart issues?
  - newfoundglory 8y ago
    This is like asking why software project estimation can't just be done by looking at the size of the project to be built.
    - emodendroket 8y ago
      I don't see how the analogy works. The entire point is you have a medical record with objective characteristics that can be tied to the risk of any given procedure.
      - newfoundglory 8y ago
        My entire point is that isn't true at all - you have a medical record with some objective characteristics, but it is not possible to calculate the risk of a procedure based on that information. Just like in software, we have a project requirements document with many objective characteristics and we are unable to calculate the time required based on those characteristics.
smsm42 8y ago
Goodhart law deals with proxies - i.e. measures not of direct target, which is hard to measure, but of something that is easy to measure, and you consider it related to the actual target. If you need wins and measure wins, wins are not a proxy and thus Goodhart law does not apply. But if, say, it were hard for you to count wins in soccer for some reason, and you measured time spent with the ball per player instead, thinking that if players own the ball all the time, they surely going to win eventually - you'd get a team that is engaged in pointless passing around instead of actually scoring goals. For surgeons, the direct target is for the patient to get better - or at least live longer and with higher quality of life. Proxy is the measure of a mortality of a specific surgeon. It is not a direct target - as the surgeon can achieve zero mortality by not doing any surgery at all, but his patients would die of suffer from the lack of treatment.
caminante 8y ago
Your question was the same question I had. The wikipedia entry didn't seem clear to me for what the law is and is not. I disagree with the downvotes your comment's received.
brightball 8y ago
In your scenario, the equivalent for win percentage would be trying to set an easier schedule.
kolbe 8y ago
Goodhart is talking about life, not a self-contained system like a game. >Some measures should be targeted, the problem here is that it is done in an non genuine way The "not in a genuine way" is exactly what he's talking about.
Godel_unicode 8y ago
Mortality rate doesn't account for the outcomes of surgeries which are declined due to risk.

Buttons840 8y ago

Pro tip for those targeting a high win percentage. If you win your first game (beginners luck?), then stop playing and never play again. That's what the article is describing, surgeons won't even play the game, they won't even attempt to help certain people because it might hurt their "win percentage".

natoliniak 8y ago
another pro tip: compete only against weaker opponents (this is common in Boxing).

washadjeffmad 8y ago

Wins are likely from systems with fixed rules that don't reward innovation, so winning by the rules of a system doesn't necessarily mean a positive outcome. An example might be the system that lead to the housing crisis. Certainly a few people who gamed things the right way "won" big, but only at the expense of millions of homeowners. Mortality rates are one measure of many that can tell a story, but they're like incarceration rates in that they've become a perverse incentive that exacerbates an issue. If we say that putting more people in prison is a good thing, that's predicated on an assumption that only those that are being put in prisons are people worthy of being there, whose freedom is more costly to society than their imprisonment. When a district is rewarded for putting people in prisons, what types of things might you expect to happen to the legal systems and populations in those places? Would prosecutors then be incentivized to trump up charges and force people into serving prison time unnecessarily to make themselves look like bigger "winners"? Might police make more frivolous arrests or even stir up trouble in communities to paint a picture of rampant criminality so they can look to be "tough on crime"? Wouldn't you expect to see an all-causes decrease in long term crime? The problem isn't just cherry picking, but if people can game a system built around limited and gameable measures, then it's going to encourage min-maxing for profit.

LoSboccacc 8y ago

the problem here is that mortality rate is not correlated/normalized with the patient risk factor. the problem lies in the performance indicator itself, not strictly people trying to optimizing for what they are asked for. it is likely inhumane to refuse patients that have slightly above average risk - even if there are some considerations to be had, like having replacement organs go to waste, but that is already covered by the donor lists queues - but given people can find themselves out of work for having bad mortality rates, optimizing for the performance indicator it's the only logical conclusion. also, normalizing for the patient risk will likely just lead to people overstating patient risk, because once you reduce performance to an index, people will focus on what's measured before on what the intended goal for the measurement is.

emodendroket 8y ago
I'm not an expert in cardiatric medicine or anything, but couldn't patient risk be determined by objective factors like patient age and history of previous heart issues?
- newfoundglory 8y ago
  This is like asking why software project estimation can't just be done by looking at the size of the project to be built.
  - emodendroket 8y ago
    I don't see how the analogy works. The entire point is you have a medical record with objective characteristics that can be tied to the risk of any given procedure.
    - newfoundglory 8y ago
      My entire point is that isn't true at all - you have a medical record with some objective characteristics, but it is not possible to calculate the risk of a procedure based on that information. Just like in software, we have a project requirements document with many objective characteristics and we are unable to calculate the time required based on those characteristics.

smsm42 8y ago

Goodhart law deals with proxies - i.e. measures not of direct target, which is hard to measure, but of something that is easy to measure, and you consider it related to the actual target. If you need wins and measure wins, wins are not a proxy and thus Goodhart law does not apply. But if, say, it were hard for you to count wins in soccer for some reason, and you measured time spent with the ball per player instead, thinking that if players own the ball all the time, they surely going to win eventually - you'd get a team that is engaged in pointless passing around instead of actually scoring goals. For surgeons, the direct target is for the patient to get better - or at least live longer and with higher quality of life. Proxy is the measure of a mortality of a specific surgeon. It is not a direct target - as the surgeon can achieve zero mortality by not doing any surgery at all, but his patients would die of suffer from the lack of treatment.

caminante 8y ago

Your question was the same question I had. The wikipedia entry didn't seem clear to me for what the law is and is not. I disagree with the downvotes your comment's received.

brightball 8y ago

In your scenario, the equivalent for win percentage would be trying to set an easier schedule.

kolbe 8y ago

Goodhart is talking about life, not a self-contained system like a game. >Some measures should be targeted, the problem here is that it is done in an non genuine way The "not in a genuine way" is exactly what he's talking about.

Godel_unicode 8y ago

Mortality rate doesn't account for the outcomes of surgeries which are declined due to risk.