10 ms·
It's Goodhart's Law in action, this time with deadly consequences: "When a measure becomes a target, it ceases to be a good measure."[a] [a] https://en.wikipe
by cs702 9y ago
It's Goodhart's Law in action, this time with deadly consequences:
"When a measure becomes a target, it ceases to be a good measure."[a]
[a] https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
- batbomb 9y agoIronically applied to not-good hearts
- truculation 9y agoIn another sense, 'Goodhart' seems like a splendid cosmic coincidence. Instead of obsessing over targets, try to perceive the whole system. And who can do that? Good-hearted individuals.
- mjfl 9y ago"These doctors are vile dogs. If I was a physician I would take all the hard patients, heal them all, and be a hero of the people." Yeah, if you were given one operation where the patient has a 100% chance of dying without it, but 80% of you killing them if you operate + 20% chance of them living after, maybe you'd do this once and get lucky. But try doing this twice, you have a a 96% chance of killing a patient, three times makes a 99.2% chance of killing at least a single patient. This probability quickly approaches 100%. If you kill a patient you get dragged into court, branded as a murderer by the opposing lawyer, and your career which you have spent your youth, your 20s and a half a million dollars training for is down the drain. Taking risky cases, if that's not your niche, is is guaranteed to have you out of a job and in debt from legal fees. To suggest that "bad hearts" are involved in this necessary calculation, is incredibly naive.
- smnrchrds 9y agoI am pretty sure that the parent was referring to the fact that this article is about heart surgeries. Bad heart is used in the same manner as a bad knee or a bad tooth. It refers to the patient not the doctor. But aside from that, I completely disagree with your comment. The approach you are defending, even if legal, is completely unethical and immoral. It should be something we chastise, not condone.
- mjfl 9y agoWell I think you're an idiot. There's only one viable "approach", determined by the laws of probability I described above. You can try to shame people into doing actions that would ultimately get them fired and sued, but you're only going to look like a self-righteous asshole and nothing will happen.
- point78 9y agoYet another reason (among many) the US is such a backwards country.
- Retric 9y agoPeople die in surgery all the time, it's not nearly as big a deal for the surgeon as you're suggesting. ~5% chance of death within 30 days per major hart surgery means the average surgeon is losing several people in an average year.
- kazinator 9y agoLow mortality is a good damned measure. I don't want to be operated on some yahoo surgeon who unconditionally operates regardless of risk and has a high mortality track record as a result.
- abritinthebay 9y agoWhen the option is "or die" I think you'd change your tune.
- sh4z 9y agoA doctor might have a 5% mortality track record for a surgery that generally has a 20% mortality risk. I feel like you are pushing it to an extreme with this yahoo doctor example.
- henriquemaia 9y agoFinger ----------------> Moon ^ You're looking at this.
- croon 9y agoThen maybe mortality rate should be attributed to doctors not operating as well. Low mortality is a good metric on patients, not doctors. But on the other hand, I don't want to be treated by a doctor who cares more about his sellable stats than saving lives.
- ScottBurson 9y agoMaybe it's a good metric for hospitals. There are other pressures discouraging hospitals from rejecting patients altogether, or such is my impression, anyway.
- dtech 9y agoIf you need a high-risk operation or die, no-one wants to operate on you because of this rule.
- venomsnake 9y agookay. You have 100% of death in next 72 hours unless there is a surgery done, which patients survive 70% of the time. No surgeon wants to do it, to not get his metric lower, so instead of 30% survival chance you get effective death sentence ...
- fc003153 9y agoI'm not so sure Goodhart's Law is correct. For example, if I target a high win percentage, is win percentage, then, not a good measure? Mortality rates are a pretty good measure for surgeons. The problem here is that they are inflating their measure by cherry picking. edit: Some measures should be targeted, the problem here is that it is done in an non genuine way
- Buttons840 9y agoPro tip for those targeting a high win percentage. If you win your first game (beginners luck?), then stop playing and never play again. That's what the article is describing, surgeons won't even play the game, they won't even attempt to help certain people because it might hurt their "win percentage".
- natoliniak 9y agoanother pro tip: compete only against weaker opponents (this is common in Boxing).
- washadjeffmad 9y agoWins are likely from systems with fixed rules that don't reward innovation, so winning by the rules of a system doesn't necessarily mean a positive outcome. An example might be the system that lead to the housing crisis. Certainly a few people who gamed things the right way "won" big, but only at the expense of millions of homeowners. Mortality rates are one measure of many that can tell a story, but they're like incarceration rates in that they've become a perverse incentive that exacerbates an issue. If we say that putting more people in prison is a good thing, that's predicated on an assumption that only those that are being put in prisons are people worthy of being there, whose freedom is more costly to society than their imprisonment. When a district is rewarded for putting people in prisons, what types of things might you expect to happen to the legal systems and populations in those places? Would prosecutors then be incentivized to trump up charges and force people into serving prison time unnecessarily to make themselves look like bigger "winners"? Might police make more frivolous arrests or even stir up trouble in communities to paint a picture of rampant criminality so they can look to be "tough on crime"? Wouldn't you expect to see an all-causes decrease in long term crime? The problem isn't just cherry picking, but if people can game a system built around limited and gameable measures, then it's going to encourage min-maxing for profit.
- waterhouse 9y agoSome guy wrote a taxonomy of Goodhart's Law cases. This would be the "Adversarial Goodhart" case: "When you optimize for a proxy, you provide an incentive for adversaries to correlate their goal with your proxy, thus destroying the correlation with your goal." https://www.lesserwrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy https://www.lesserwrong.com/posts/EbFABnst8LsidYs5Y/goodhart...
- dooglius 9y agoThe first example on that page, "Regressional Goodhart", is totally wrong. If U measures V plus some noise X, assuming V and X form a bivariate normal distribution, then the conditional expectation of V is maximized by selecting the greatest U. In fact, the conditional expectation is linear in U. Not sure how much stock I can put into the rest of the article given this. Goodhart's Law can seemingly only apply if the variables are highly non-normal, or are negatively correlated (e.g. worse doctors are more likely to refuse the difficult operations than good doctors).
- waterhouse 9y ago> The first example on that page, "Regressional Goodhart", is totally wrong. I don't think your statements contradict anything that it says. For reference, here are all the sentences in the "Quick Reference" part, which I'm assuming is the part you read. Regressional Goodhart - When selecting for a proxy measure, you select not only for the true goal, but also for the difference between the proxy and the goal. Model: When U is equal to V+X, where X is some noise, a point with a large U value will likely have a large V value, but also a large X value. Thus, when U is large, you can expect V to be predictably smaller than U. Example: height is correlated with basketball ability, and does actually directly help, but the best player is only 6'3", and a random 7' person in their 20s would probably not be as good Is any particular one of these sentences false? > If U measures V plus some noise X, assuming V and X form a bivariate normal distribution, then the conditional expectation of V is maximized by selecting the greatest U. The word "maximized" carries some assumptions. If the only thing you can do is select based on U, then that statement is correct. However, if you had some means of selecting directly on V, then it's extremely likely that this would do better than selecting on U. Suppose you're selecting the top 10 people. Suppose X ranges 0-10 chosen by a die roll, and V ranges 0-10, and it happens there are ten people with V=10, a hundred people with V=9, and a thousand with V=8 (and a lot more with lower Vs). On average, you'll have ten V=9s who score U=19, ten V=9s and a hundred V=8s who score U=18, and so on; in order for selecting on U to perform as well as selecting on V, every one of the V=10s would have to roll X=9 or X=10, which is exceedingly unlikely. It is true that taking people with high U scores yields people with higher Vs than taking people at random, and it is further true that taking the U=19s will give you better results than taking the U=18s or U=12s. It is simultaneously true that, when you take the U=19s, you'll be getting people whose X was 9 or 10, much higher than if you selected people at random or if you selected directly for high V. The first two sentences from the text state exactly this. (One consequence of this observation is, e.g., if you're doing admissions based on some test score, and you're considering raising the required score by ∆U, you should know the effect will be to raise average V and to raise average X, with ∆V=∆U-∆X, and if ∆X is large, you may be disappointed in the results. This is simple regression to the mean.) I imagine you know all these concepts; I think you're interpreting the text as a stronger statement than it is.
- JulianMorrison 9y agoThat only really applies when you're measuring side effects though, and not the actual thing intended. In this case they should have had a metric that took into account the estimated prior probability of a good outcome given an ordinary surgeon.
- dfox 9y agoThe problem is that in many cases, including this, you cannot directly measure the intended thing in any meaningful way. Your example is just another indirection which is still certainly somewhat playable and in addition involves estimation from insufficient input data (also known as guessing ;)).
- contravariant 9y agoMortality rate is a side effect. Survival is the intended effect, but that's much more of a binary thing.
- olliej 9y agoIt also kind of incentivizes the alternative as well: if you aren’t an amazing surgeon, you could work to ensure you exclusively work on “hard” cases you can claim all failures are due to patient difficulty rather than lack of competence (at least competence to deal with that difficulty). Then you can dismiss any claims you’re incompetent with “I have a high mortality rate because I’m willing to risk my reputation to save people who have been abandoned”...
- Arnt 9y agoI've listened to a physician make this argument (for a different metric, not mortalities). He sounded as though he expected everyone in the audience to disbelieve him, enough so that I wondered why he even tried. So yes, could.
- emodendroket 9y agoWhen I was fresh out of a school I worked in a call center for a while. One of the most important metrics (and everything you did was measured) was the average length of each call. So it was common practice, when you got cases that basically were going to be long calls or ones you probably couldn't actually resolve (e.g., "I want the service back on but cannot pay"), to deliberately upset the customers so that they would demand a supervisor and you could transfer them.
- smsm42 9y agoI'd also expect a bunch of "I'm going to transfer you... oops, pressed a wrong button and it hang up!" As a customer, I've had a fair measure of mysteriously hang up calls to support, I wonder how many of those are metrics-driven...
- emodendroket 9y agoYeah, eventually they started measuring transfers as a metric, which indeed encouraged that behavior. Of course some people would just say the issue was solved, falsely (which, you know, it was a little easier to get caught so it was higher-risk, in addition to being probably less moral), and there was also the X factor of how much you liked the customer, considering the extreme amount of verbal abuse the job entailed.
- pmoriarty 9y agoThere was recently a story about a 911 operator being jailed for hanging up on thousands of calls.[1] If someone could get away with it at a 911 call center, it could happen anywhere... and might happen oftener than we think. [1] - https://www.sfgate.com/news/article/911-dispatcher-sentenced-to-jail-after-hanging-up-12847942.php https://www.sfgate.com/news/article/911-dispatcher-sentenced...
- briandear 9y agoThat Houston lady that did that didn’t care about metrics, she literally just couldn’t be bothered. “Ain’t nobody got time for that.”
- iovrthoughtthis 9y agoCame here to say this.
- sowbug 9y agoThe version I've heard might be a corollary: if you measure it, it will improve. The subtext is that the system won't improve; rather, something in the system will be sacrificed to improve the metric. The end result (usually quality) won't improve.
- danzig13 9y agoAt best it probably only identifies the people to replace.
- danzig13 9y agoThanks. A few years ago I asked a question on what this was called.
- lambdasquirrel 9y agoOn a less dire area of work, I’m starting to turn away from risky jobs/startups. It seems like there’s more and more newcomers in tech these days, and they’ll reject you for not being at gigs for longer. If I get the impression a future manager will be inexperienced, or that they’d failed to build a team before, I hesitate now. It’s a little frustrating, but that’s just how it is.
- sp527 9y agoAs a corollary, this is along the lines of one of my favorite bits of wisdom from Poor Charlie's Almanac. If I recall correctly, Munger considers incentives to be the single most important concept to properly understand in order to drive successful business (and arguably life) outcomes. Not surprisingly, incentives are also chronically underestimated or outright ignored, even in situations where there's a strong, profit-driven incentive (#meta) to get them right. > From all business, my favorite case on incentives is Federal Express. The heart and soul of their system – which creates the integrity of the product – is having all their airplanes come to one place in the middle of the night and shift all the packages from plane to plane. If there are delays, the whole operation can’t deliver a product full of integrity to Federal Express customers. And it was always screwed up. They could never get it done on time. They tried everything – moral suasion, threats, you name it. And nothing worked. Finally, somebody got the idea to pay all these people not so much an hour, but so much a shift and when it’s all done, they can all go home. Well, their problems cleared up overnight.
- pmoriarty 9y agoFor a different view, see "Punished by Rewards: The Trouble with Gold Stars, Incentive Plans, A's, Praise, and Other Bribes".[1] It's about extrinsic vs intrinsic motivation. Extrinsic motivation comes from outside of the person and is what's typically used by organizations to try to motivate people: salary, praise, bonuses, etc. Intrinsic motivation comes from within. [1] - https://www.amazon.com/Punished-Rewards-Trouble-Incentive-Praise/dp/0618001816/ https://www.amazon.com/Punished-Rewards-Trouble-Incentive-Pr...
- Godel_unicode 9y agoI think this is frequently short-sighted. Surgeons are intrinsically motivated to be surgeons and help people, and they are trained to think clinically (pun somewhat intended) about risk-reward. They realize that trying to help high-risk patients will ultimately result in them being able to help fewer patients.
- candiodari 9y agoLuckily in most businesses incentives are used by the people designing the incentives to 1) cost as little as possible to the business 2) "optimize" their own pay/career (There was a study saying that over 80% of sales people would deny their employer a million-dollar contract if it netted them $500 personally, so compared to that this is nothing) Which makes sure that the incentives are not aligned with the business goals. Now I guarantee that the best way to destroy intrinsic motivation, bar none, is to provide extrinsic motivation for things that aren't aligned with the business' goals.
- simonebrunozzi 9y agoApologies for the silly comment in advance - but how ironical that "Goodhart" is applied in this case to heart surgeons.
- hliyan 9y agoI prefer to think that this happens when you pick a measure that ignores key aspects of the results. For example, would the results be the same if the mortality rate was complexity-weighted?