3 ms·

One thing I see far too rarely in choosing metrics is ensuring that every metric has one or more countervailing metrics to balance the choices made. In other w

One thing I see far too rarely in choosing metrics is ensuring that every metric has one or more countervailing metrics to balance the choices made. In other words, if you are optimizing for a metric, you also need to measure what is being sacrificed for it. The classic trio of cheap/fast/good is a great example of this, as are most engineering-style tradeoffs, like capacity vs range. You will often see managers create metrics for the things they want to improve, without realizing that making a metric without measuring the countervailing metric is creating an incentive to throw the countervailing metric in the shitter. A good example of this is call centers which measure the time of call resolution without measuring customer satisfaction with the call. This is how you end up with a low quality, high volume call center. You can do the opposite by measuring only satisfaction and not speed. Now, maybe what your business needs is high volume, low quality, but that should be a choice you make. The problem with not measuring the countervailing metric is that nobody (officially) cares about how bad that metric gets. So, if you have a tradeoff situation, you automatically get an extreme even though that may not be the most profitable place on the spectrum. Perhaps your call center would be better for the company if it was volume 9 quality 2 instead of volume 10 quality 1, but if you only optimize for volume you can't make that choice. This even applies to metrics you really want to go up, things like sales and profit. A manufacturing company that optimizes only for sales while ignoring manufacturing metrics will hire salespeople in preference to manufacturing workers until the inability to deliver product starts affecting sales. That might sound like a nice problem to have, but it can be a real problem, and perhaps sales 9 manufacturing 2 instead of sales 10 manufacturing 1 would result in better sales next year. It's better to have metrics which allow you that choice.

hbrn 4y ago
That's basically recognizing that Goodhart's Law exists and fighting it. Unfortunately managers rarely have incentives to do it. Why invest in being more objective if it lowers the perceived value of your successes? Your bullshitting peer that isn't doing this will get promoted instead of you. It can also be hard to even understand what are you sacrificing and why. I recently had an example where product manager was trying to improve conversion while another team was tapping into cheap low-quality traffic. PM always looked at a global conversion and kept wondering why the conversion was dropping despite his best efforts.
- random314 4y ago
  Absolutely true. Managers absolutely dont want countervailing metrics. I noticed a huge problem at a FAANG in a 40 person team and proposed the design of a single metric that incorporated all costs including countervailing metrics. There were no takers for the project. The senior manager leading 4 teams of 40 engineers told me privately. Your proposal is relatively cheap to implement and I can't ask for more HC for this and doesn't help me with empire building. Your metrics will make the team achievements look smaller and weaker. Your metrics will anger many of the teams, because the team switches the metric to target each half. In your example, targeting customer throughput one half and customer satisfaction the next half thus running a merry go round. The team has become comfortable switching metrics each half and it is simple work. If your metric is set as a target, the engineering problem becomes a lot harder and ICs will be upset. So, my proposal was canned.
  - water554 4y ago
    That was nice of him to share that with you.
    - random314 4y ago
      Well, he didn't call it empire building himself :D That's my word. And I have been working with him for several years already, so discussions are frank! TBF, it can be a justifiable POV from a /r/antiwork POV. Why work terrible hours to make someone else rich!

hbrn 4y ago

That's basically recognizing that Goodhart's Law exists and fighting it. Unfortunately managers rarely have incentives to do it. Why invest in being more objective if it lowers the perceived value of your successes? Your bullshitting peer that isn't doing this will get promoted instead of you. It can also be hard to even understand what are you sacrificing and why. I recently had an example where product manager was trying to improve conversion while another team was tapping into cheap low-quality traffic. PM always looked at a global conversion and kept wondering why the conversion was dropping despite his best efforts.

random314 4y ago
Absolutely true. Managers absolutely dont want countervailing metrics. I noticed a huge problem at a FAANG in a 40 person team and proposed the design of a single metric that incorporated all costs including countervailing metrics. There were no takers for the project. The senior manager leading 4 teams of 40 engineers told me privately. Your proposal is relatively cheap to implement and I can't ask for more HC for this and doesn't help me with empire building. Your metrics will make the team achievements look smaller and weaker. Your metrics will anger many of the teams, because the team switches the metric to target each half. In your example, targeting customer throughput one half and customer satisfaction the next half thus running a merry go round. The team has become comfortable switching metrics each half and it is simple work. If your metric is set as a target, the engineering problem becomes a lot harder and ICs will be upset. So, my proposal was canned.
- water554 4y ago
  That was nice of him to share that with you.
  - random314 4y ago
    Well, he didn't call it empire building himself :D That's my word. And I have been working with him for several years already, so discussions are frank! TBF, it can be a justifiable POV from a /r/antiwork POV. Why work terrible hours to make someone else rich!