3 ms·

That is partially true. A metric should continue to "work" when people know they are being measured on it. A good metric is one which makes the measured person

by ken47 12y ago

That is partially true. A metric should continue to "work" when people know they are being measured on it. A good metric is one which makes the measured person more effective when they explicitly try to optimize it. So for example, pro basketball player knows they are measured by their accuracy. However, when a player optimizes his accuracy, that does not compromise the effectiveness of that metric.

bzbarsky 12y ago
The important thing here is that a metric that is a _correlate_ of the thing you care about but can't measure directly might start to work less well once people know you're measuring it. And the reason is that the correlation with the thing you really care about might decrease. To use your basketball example, say you have a player that takes a shot every time they get close enough to the basket, scoring 60% of them. You decide to measure accuracy. They get more conservative with when they take shots, so now half the time they run out of shot clock (resulting in a turnover) and the other half of the time they take a shot and score with 100% accuracy. Their accuracy went from 60% to 100%, but the fraction of time possession results in scoring went from 60% to 50%. The issue there is that you don't _really_ care about accuracy per se. What you care about is contribution to the score differential vs the opposing team. But that's hard to quantify, so people come up with various metrics that correlate to such contribution (shooting accuracy, rebound percentage, etc, etc), along with relative weights for these metrics. In fact, the weights depend on the exact team you're playing, who your current teammates are, etc. In any case, it's actually fairly rare to be able to directly measure the thing you care about, especially in the social sciences, not least because a lot of the things we care about ("success in life", "college preparedness", "job satisfaction", "knowledge and understanding of the material covered by this class") are fairly difficult to quantify. So instead one ends up measuring various correlates and hoping that either people don't adjust their behavior too much to degrade the correlation before you're done measuring or that your measurement process attempts to account for such adjustments.
- mathattack 12y ago
  I was thinking similarly. If you pay salespeople only on Sales $, it sounds like incentives are aligned, but you have to watch yourselves.
  - IkmoIkmo 12y ago
    Indeed. Just yesterday a buddy told me of how he knew someone a few years ago (great story I know) who was gaming telephone subscriptions. He'd go into the system and check a box, say 'extra 500 minutes' with a deal that gave the first 3 months for free. This is normally what you do after you close such a sale when a customer buys the extra deal, you go into the system and register it. Only now he'd just do it without the customer knowing. 3 months later he'd cancel it. His sale was still registered and his customer never knew or ever paid, and he got his sales bump. And another friend of mine works at a big software company (SAP, Oracle, those sorts of companies) in sales. Basically he has to generate leads, pass them on, and if they're closed he hits his target. So what most people do is they call the manager and suck up to him, and ask him to register a closed sale's lead as one of his. A lot of these sales are simply recurring sales, or sales closed when a customer called the company, as opposed to closed after a cold-call led to a lead. Long story short, you've got people hitting their targets without generating any leads or sales, making $40k a year and moving into the same closing function two years later without ever having done anything. The idea seems pretty straightforward, once you create a metric, people will game it. And it seems a single person designing the system (the metrics, the governance of the metrics) over time continuously gets outsmarted by the hundreds or even thousands of people that are incentivized to find the loophole in the design of that individual.

by ken47 12y ago

bzbarsky 12y ago

The important thing here is that a metric that is a _correlate_ of the thing you care about but can't measure directly might start to work less well once people know you're measuring it. And the reason is that the correlation with the thing you really care about might decrease. To use your basketball example, say you have a player that takes a shot every time they get close enough to the basket, scoring 60% of them. You decide to measure accuracy. They get more conservative with when they take shots, so now half the time they run out of shot clock (resulting in a turnover) and the other half of the time they take a shot and score with 100% accuracy. Their accuracy went from 60% to 100%, but the fraction of time possession results in scoring went from 60% to 50%. The issue there is that you don't _really_ care about accuracy per se. What you care about is contribution to the score differential vs the opposing team. But that's hard to quantify, so people come up with various metrics that correlate to such contribution (shooting accuracy, rebound percentage, etc, etc), along with relative weights for these metrics. In fact, the weights depend on the exact team you're playing, who your current teammates are, etc. In any case, it's actually fairly rare to be able to directly measure the thing you care about, especially in the social sciences, not least because a lot of the things we care about ("success in life", "college preparedness", "job satisfaction", "knowledge and understanding of the material covered by this class") are fairly difficult to quantify. So instead one ends up measuring various correlates and hoping that either people don't adjust their behavior too much to degrade the correlation before you're done measuring or that your measurement process attempts to account for such adjustments.

mathattack 12y ago
I was thinking similarly. If you pay salespeople only on Sales $, it sounds like incentives are aligned, but you have to watch yourselves.
- IkmoIkmo 12y ago
  Indeed. Just yesterday a buddy told me of how he knew someone a few years ago (great story I know) who was gaming telephone subscriptions. He'd go into the system and check a box, say 'extra 500 minutes' with a deal that gave the first 3 months for free. This is normally what you do after you close such a sale when a customer buys the extra deal, you go into the system and register it. Only now he'd just do it without the customer knowing. 3 months later he'd cancel it. His sale was still registered and his customer never knew or ever paid, and he got his sales bump. And another friend of mine works at a big software company (SAP, Oracle, those sorts of companies) in sales. Basically he has to generate leads, pass them on, and if they're closed he hits his target. So what most people do is they call the manager and suck up to him, and ask him to register a closed sale's lead as one of his. A lot of these sales are simply recurring sales, or sales closed when a customer called the company, as opposed to closed after a cold-call led to a lead. Long story short, you've got people hitting their targets without generating any leads or sales, making $40k a year and moving into the same closing function two years later without ever having done anything. The idea seems pretty straightforward, once you create a metric, people will game it. And it seems a single person designing the system (the metrics, the governance of the metrics) over time continuously gets outsmarted by the hundreds or even thousands of people that are incentivized to find the loophole in the design of that individual.