7 ms·

Humans have proven time and time again that we are very good at optimizing for specific metrics, and apparently generally very poor at choosing the right metric

by ar_lan 6y ago

Humans have proven time and time again that we are very good at optimizing for specific metrics, and apparently generally very poor at choosing the right metrics to target. School is a prime example, in my opinion - there is so much I "learned" in school that I'll never remember, because I didn't take a proper approach to learning - I just optimized for grades. I had a 4.7 GPA and could hardly tell you a thing about US history or recite a lick of Spanish, because I simply did not care about anything but getting the A so I could get into the college I cared about. In the work environment, I can see a lot of parallels to this. If I'm competing with my coworkers, my incentive is to outwork and outshine them. A common thing I see is when a coworker does a bit of innovative work - it's almost guaranteed some other coworker will intercede before a chance at applause is given to discuss its obvious flaws, the plethora of alternatives out there, etc., leading the developer to feel like their implementation was not good. (For the record, I'm not talking about general criticism - I'm talking about the general pattern of not celebrating someone's achievements and then discussing how to continue improving, but rather a "why did you even do this" mentality).

biswaroop 6y ago
> Humans have proven time and time again that we are very good at optimizing for specific metrics, and apparently generally very poor at choosing the right metrics to target. Maybe it's Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."
- jonpurdy 6y ago
  As another example: Agile dev team measuring velocity (story points per sprint) to predict how much work they can do in the future. If velocity gets turned into a target, corners will be cut (less testing, poor code quality) in order to get those story points for that sprint. Code quality suffers and tech debt accumulates, reducing output down the road. On the flip side, as time goes on teams could just progressively estimate a greater number of points to stories of similar difficulty during the planning process. So velocity goes up but actual deliverable features goes down or stays the same.
- shellum 6y ago
  I agree that it is easy for a metric to loose it's original meaning. I've had luck with OKRs; Having a non quantitative goal, and possibly changing quantitative based targets aimed at making the goal happen.
JMTQp8lwXL 6y ago
If we don't have metrics, how are we to fairly and objectively compare one worker's performance to another?
- btowngar 6y ago
  It's not that metrics in general are bad, it's that bad metrics are bad.
  - shredprez 6y ago
    What's the best metric for metric quality?
    - jonpurdy 6y ago
      It's metrics all the way down...
- amznthrowaway5 6y ago
  How is obsessive focus on metrics making anything more fair? Even if you have metrics, choosing which ones are important still ends up being subjective, and the metrics are usually incomparable between projects. So the metrics alone don't tell anything useful and can still be easily gamed to make the manager's preferred employees look good and disliked employees look bad.
  - andreilys 6y ago
    Metrics are CYA insurance for managers to promote or PIP employees without getting into trouble with HR
- milesvp 6y ago
  Let me ask you another question. How do you identify the additive employees from the multiplicative employees? You probably don’t want all multiplicative employees on a team, but you surely want some of them. How do you create metrics that don’t encourage too much of one or the other? How much is too much of one or the other?
yreg 6y ago
Planet Money had an episode on such metrics[0] The host shares a personal story from time when he was a cashier. The grocery chain measured performance as items scanned per minute and shared internal leaderboard. Some items are hard to scan so the cashiers ended up skipping scanning them – so the store gave them away for free. But the managers were happy about their KPIs. (Look how many items are we suddenly selling per minute!) [0] - https://www.npr.org/transcripts/669396192 https://www.npr.org/transcripts/669396192 (both podcast and transcript) I love this podcast.
- Cerium 6y ago
  Costco has such leader boards and keeps a separate board for part timers vs full time cashiers. I asked my friend who works there why the part time cashiers have a much higher (upwards of 2x) items per minute rate than the full time employees. Costco only schedules part time cashiers during busy hours where the lines are long and there are assistants getting the carts ready.
- cpeterso 6y ago
  I heard about a call center that rewarded customer service agents for short calls, so workers started immediately hanging up on some calls to bring their average call duration down.
  - bombledmonk 6y ago
    I can verify this does happen in some call centers.
  - throwaway987978 6y ago
    I swear this happened to me with Grubhub. I had a problem with my order and used their chat. After I wrote my problem the agent replied with a standard answer and then immediately: "Is there anything else I can help you with?" And then immediately after that: "No, ok I will close this chat."
  - RotANobot 6y ago
    > I heard about a call center that rewarded customer service agents for short calls Almost all call centres reward agents for short calls.
A4ET8a8uTh0 6y ago
One cannot overstate this and human capability to optimize for metrics ( and what is apparently rewarded ). Anecdote time. My former manager had an idea to measure the amount of average alerts per hour and tore into people, whose values fell below a certain treshold. What did people do? They stopped doing harder alerts whenever they could and did 'easy ones' to pump up their stats. I have seen it since and whenever I do, I have a quick talk about people not being idiots and importance of metrics not being used for evaluation ( or if you do, to be ready for its consequences ). In case you are wondering, former's managers response was to double down and hide the stats so that people don't know where they stand.. eh.
- amatecha 6y ago
  "When a measure becomes a target, it ceases to be a good measure".. I think this phrase all too often... haha
ausjke 6y ago
no better alternatives, if you do not do grading etc at school then what we have is a small group of kids learned a lot on their own(1% of them I guess), the rest 99% will just waste their youth totally and learned nothing at all.

by ar_lan 6y ago

biswaroop 6y ago

> Humans have proven time and time again that we are very good at optimizing for specific metrics, and apparently generally very poor at choosing the right metrics to target. Maybe it's Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."

jonpurdy 6y ago
As another example: Agile dev team measuring velocity (story points per sprint) to predict how much work they can do in the future. If velocity gets turned into a target, corners will be cut (less testing, poor code quality) in order to get those story points for that sprint. Code quality suffers and tech debt accumulates, reducing output down the road. On the flip side, as time goes on teams could just progressively estimate a greater number of points to stories of similar difficulty during the planning process. So velocity goes up but actual deliverable features goes down or stays the same.
shellum 6y ago
I agree that it is easy for a metric to loose it's original meaning. I've had luck with OKRs; Having a non quantitative goal, and possibly changing quantitative based targets aimed at making the goal happen.

JMTQp8lwXL 6y ago

If we don't have metrics, how are we to fairly and objectively compare one worker's performance to another?

btowngar 6y ago
It's not that metrics in general are bad, it's that bad metrics are bad.
- shredprez 6y ago
  What's the best metric for metric quality?
  - jonpurdy 6y ago
    It's metrics all the way down...
amznthrowaway5 6y ago
How is obsessive focus on metrics making anything more fair? Even if you have metrics, choosing which ones are important still ends up being subjective, and the metrics are usually incomparable between projects. So the metrics alone don't tell anything useful and can still be easily gamed to make the manager's preferred employees look good and disliked employees look bad.
- andreilys 6y ago
  Metrics are CYA insurance for managers to promote or PIP employees without getting into trouble with HR
milesvp 6y ago
Let me ask you another question. How do you identify the additive employees from the multiplicative employees? You probably don’t want all multiplicative employees on a team, but you surely want some of them. How do you create metrics that don’t encourage too much of one or the other? How much is too much of one or the other?

yreg 6y ago

Planet Money had an episode on such metrics[0] The host shares a personal story from time when he was a cashier. The grocery chain measured performance as items scanned per minute and shared internal leaderboard. Some items are hard to scan so the cashiers ended up skipping scanning them – so the store gave them away for free. But the managers were happy about their KPIs. (Look how many items are we suddenly selling per minute!) [0] - https://www.npr.org/transcripts/669396192 https://www.npr.org/transcripts/669396192 (both podcast and transcript) I love this podcast.

Cerium 6y ago
Costco has such leader boards and keeps a separate board for part timers vs full time cashiers. I asked my friend who works there why the part time cashiers have a much higher (upwards of 2x) items per minute rate than the full time employees. Costco only schedules part time cashiers during busy hours where the lines are long and there are assistants getting the carts ready.
cpeterso 6y ago
I heard about a call center that rewarded customer service agents for short calls, so workers started immediately hanging up on some calls to bring their average call duration down.
- bombledmonk 6y ago
  I can verify this does happen in some call centers.
- throwaway987978 6y ago
  I swear this happened to me with Grubhub. I had a problem with my order and used their chat. After I wrote my problem the agent replied with a standard answer and then immediately: "Is there anything else I can help you with?" And then immediately after that: "No, ok I will close this chat."
- RotANobot 6y ago
  > I heard about a call center that rewarded customer service agents for short calls Almost all call centres reward agents for short calls.

A4ET8a8uTh0 6y ago

One cannot overstate this and human capability to optimize for metrics ( and what is apparently rewarded ). Anecdote time. My former manager had an idea to measure the amount of average alerts per hour and tore into people, whose values fell below a certain treshold. What did people do? They stopped doing harder alerts whenever they could and did 'easy ones' to pump up their stats. I have seen it since and whenever I do, I have a quick talk about people not being idiots and importance of metrics not being used for evaluation ( or if you do, to be ready for its consequences ). In case you are wondering, former's managers response was to double down and hide the stats so that people don't know where they stand.. eh.

amatecha 6y ago
"When a measure becomes a target, it ceases to be a good measure".. I think this phrase all too often... haha

ausjke 6y ago

no better alternatives, if you do not do grading etc at school then what we have is a small group of kids learned a lot on their own(1% of them I guess), the rest 99% will just waste their youth totally and learned nothing at all.