3 ms·

Whenever metrics are instrumented that measure productivity, safety, quality, or efficiency, the metrics and metrics alone are what matter, not the over arching

Whenever metrics are instrumented that measure productivity, safety, quality, or efficiency, the metrics and metrics alone are what matter, not the over arching goal. Whether it's car crashes, fuel efficiency, lines of code, completed tickets, OKRs, or anything else, players in the system adjust to optimize output for the evaluated metric. A tale as old as time.

spott 6y ago
Unfortunately, there isn't really a solution to this problem. Metrics are the only way to measure some things reliably and consistently across users, testers and subjects. The goal is to have performance on the metric be as close as possible to the desired characteristic. This isn't always possible (and gets really hard when people are involved), but there isn't really any other solution.
- nitrogen 6y ago
  Hidden metrics help, but it is pretty frustrating as an individual to be evaluated based on hidden criteria.
  - spott 6y ago
    Yea, and no metric stays hidden forever if there is enough money in it getting unhidden. Statistical metrics have some of the benefits of hidden metrics while staying "hidden" forever, but they also can be much more time consuming to do right and are even more frustrating to be evaluated by.
csours 6y ago
The more direct control people have of a metric, the more likely it is to be gamed. Some metrics are almost guaranteed to be meaningless and unproductive: bug counts (open/closed), lines of code, closed tickets. Other metrics are informative, but game-able: fuel efficiency, crash tests. Other metrics are expensive and potentially confusing: customer satisfaction, number of automotive deaths, CO2 levels, pollution levels. If you are measuring a game-able metric, you should probably also measure a higher level metric and make sure that they are both heading in the same direction. Obviously this is expensive and complicated to figure out.

spott 6y ago

Unfortunately, there isn't really a solution to this problem. Metrics are the only way to measure some things reliably and consistently across users, testers and subjects. The goal is to have performance on the metric be as close as possible to the desired characteristic. This isn't always possible (and gets really hard when people are involved), but there isn't really any other solution.

nitrogen 6y ago
Hidden metrics help, but it is pretty frustrating as an individual to be evaluated based on hidden criteria.
- spott 6y ago
  Yea, and no metric stays hidden forever if there is enough money in it getting unhidden. Statistical metrics have some of the benefits of hidden metrics while staying "hidden" forever, but they also can be much more time consuming to do right and are even more frustrating to be evaluated by.

csours 6y ago

The more direct control people have of a metric, the more likely it is to be gamed. Some metrics are almost guaranteed to be meaningless and unproductive: bug counts (open/closed), lines of code, closed tickets. Other metrics are informative, but game-able: fuel efficiency, crash tests. Other metrics are expensive and potentially confusing: customer satisfaction, number of automotive deaths, CO2 levels, pollution levels. If you are measuring a game-able metric, you should probably also measure a higher level metric and make sure that they are both heading in the same direction. Obviously this is expensive and complicated to figure out.