5 ms·

Plato's Dashboards

61 points by eproxus 5y ago

hinkley 5y ago
> So let's circle back to metrics and ways to ensure we use them for guidance rather than obey them reflexively. I'm not sure this is the right framing of the problem. "Guidance" is too nebulous and leaves us with the same problem of interpretation. Advice from an old tracker: You want to find someone, use your eyes. When I have someone who is getting themselves/us in trouble by looking at charts, my remedy always includes the notion that charts are for asking questions, not answering them. You might have a chart that says user response time has improved, but have you tried using the service from your phone (with wifi disabled)? If the chart says the times are shifting, we need to know why they are shifting, because that's information. "Looks like response time went down," is only useful as the introduction to a question, like, "did we change something?" or "did we have a routing issue?", or "can we do the same thing on this other service that is drawing complaints?" Getting good questions also often involves cleaning up sources of noise in the charts. It's too easy for someone with an axe to grind to jump to conclusions that this is a repeat of a problem we had before (ie, this is Team X's fault... again) or that lack of improvement means failure to act. It is not impossible that I fix one regression at the same time someone else causes another, or changes the denominator by the same magnitude as the fix.
throwaway2331 5y ago
I'll barge in and say that there's this meme in business schools that: "what gets measured, gets done." They're mostly there to serve as the source of truth, whether or not a specific manager is getting a bonus (and by how much). Generally, the best metrics are "based in reality" (and usually revolve around inflow and outflow of money) -- but those are hard to fudge, without playing accounting games, and even harder to meet (it takes actual skill, rather than being able to pass the buck endlessly for a couple of years, before moving onto another org); so you may find it common for the managerial horde to pick "bullshit metrics" to cover their own asses (doubly so, if the metrics are decided by committee). There's no reason for practical metrics to exist in most large corporations -- the incentives just aren't aligned. Bonuses are decided by metrics (and anyone who has a modicum of intuition, will always pick the metric that can be most gamed for himself, and his underlings). Also, if we're being real with ourselves, there's very little that managers "control," so picking a "real" metric would just breed an environment that selects for luck (or lies). In my opinion, the only useful metric is profit generated. Everything else is just an emotional safety blanket for uncertainty.
- marginalia_nu 5y ago
  > In my opinion, the only useful metric is profit generated. Everything else is just an emotional safety blanket for uncertainty. This too can be a toxic metric. Only focusing on what looks good in the quarterly report encourages a type of short-sightedness that has been the undoing of many businesses.
  - zaidf 5y ago
    In my opinion, the only useful metric is profit generated We know by observing private equity that the easiest way to hack this metric is by doing layoffs until there is imminent threat of the business vanishing.
    - dsr_ 5y ago
      It's also not observable in real-time; often only quarterly or annually. That makes it really hard to make tactical decisions.
  - throwaway2331 5y ago
    Yes; it's a trade-off due to how difficult it is to make people give a shit about the short-term---forget about the long-term (if there's no stock involved. We've barely figured out how to make CEOs tuned into the "long-term"). The corp is just a vehicle for the goals of all its passengers. Most of them are only on-board for a couple of stops, and then off to the next vehicle. To voluntarily forgo getting off at your stop, without receiving something of equal or greater value, seems awfully foolish.
- cconstantine 5y ago
  The only thing that matters is revenue, and if it can't be easily measured against revenue it doesn't exist. I am so tired of this line of reasoning. It's poisoning the industry, ruining products, and sucking the joy out of work. What about making a quality product because you have a passion for it? Nope, make a broken MVP and move on to the next customer. If I can't maintain a metric until I can tie it to revenue, then I guess I don't get cpu usage on our database. Or disk usage. Or response time for anything that only employees touch. Bad data and errors are fine as long as no one complains. Actually, if we produce a shitty product long enough everyone will get used to it and stop complaining. If we stop measuring error rates we can stop wasting time on fixing things right? Tests and code refactors just get in the way of delivering that MVP. Rewards are doled out for fixing an outage, and preventing them is a cost-center to be eliminated. On the flip side, now we've got KPIs for our ability to push things through the corporate bureaucracy. I didn't become a programmer so I can waste time estimating how long it'll take to produce a design doc that'll be useless by the time I get to implementation, just so a bureaucrat can decide if some deeply technical problem they don't understand should be fixed. There has to be a better way.
phillipcarter 5y ago
I don't have much to add other than this reminds me of the obsession around "satisfaction" for product metrics and why that seems to always boil down to slapping NPS on it because it's a number and execs like numbers. I think this is essential reading if anyone's interested on a product-focused tangent for the topic: https://articles.uie.com/net-promoter-score-considered-harmful-and-what-ux-professionals-can-do-about-it/ https://articles.uie.com/net-promoter-score-considered-harmf...
- splittingTimes 5y ago
  Sorry to ask but What's NPS?
  - cconstantine 5y ago
    Net Promoter Score: https://en.wikipedia.org/wiki/Net_promoter_score https://en.wikipedia.org/wiki/Net_promoter_score How likely are people to promote the thing?
- malshe 5y ago
  I really liked this article. Thanks for sharing!
deleted 5y ago
[deleted]
a9h74j 5y ago
FWIW [from having skimmed] This should be of interest to people interested in UX where there are safety ramifications. Thus, the original article title might be a bit misleading. It is not necessarily about businesses running based upon illusions.
ngcc_hk 5y ago
“I'll barge in and say that there's this meme in business schools that: "what gets measured, gets done."” The key is those that sone might not be most useful.
tpoacher 5y ago
Goodhart's law is often confused with Campbell's law. The two are indeed very similar, and often used interchangeably, but there's an important difference in their phrasing. Goodhart's law states that: "When a measure becomes a target, it ceases to be a good measure." Campbell's law states that: "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." In other words, the former focuses on the fact that, while metrics sound good in theory, in practice they cannot be relied on; whereas the latter focuses on the fact that enforcing metrics in an attempt to control a social process tends to backfire. Or, in Trump language: Goodhart = "Sounds good, doesn't work" Campbell = "Worst trade deal ever"

61 points by eproxus 5y ago

> So let's circle back to metrics and ways to ensure we use them for guidance rather than obey them reflexively. I'm not sure this is the right framing of the problem. "Guidance" is too nebulous and leaves us with the same problem of interpretation. Advice from an old tracker: You want to find someone, use your eyes. When I have someone who is getting themselves/us in trouble by looking at charts, my remedy always includes the notion that charts are for asking questions, not answering them. You might have a chart that says user response time has improved, but have you tried using the service from your phone (with wifi disabled)? If the chart says the times are shifting, we need to know why they are shifting, because that's information. "Looks like response time went down," is only useful as the introduction to a question, like, "did we change something?" or "did we have a routing issue?", or "can we do the same thing on this other service that is drawing complaints?" Getting good questions also often involves cleaning up sources of noise in the charts. It's too easy for someone with an axe to grind to jump to conclusions that this is a repeat of a problem we had before (ie, this is Team X's fault... again) or that lack of improvement means failure to act. It is not impossible that I fix one regression at the same time someone else causes another, or changes the denominator by the same magnitude as the fix.

throwaway2331 5y ago

I'll barge in and say that there's this meme in business schools that: "what gets measured, gets done." They're mostly there to serve as the source of truth, whether or not a specific manager is getting a bonus (and by how much). Generally, the best metrics are "based in reality" (and usually revolve around inflow and outflow of money) -- but those are hard to fudge, without playing accounting games, and even harder to meet (it takes actual skill, rather than being able to pass the buck endlessly for a couple of years, before moving onto another org); so you may find it common for the managerial horde to pick "bullshit metrics" to cover their own asses (doubly so, if the metrics are decided by committee). There's no reason for practical metrics to exist in most large corporations -- the incentives just aren't aligned. Bonuses are decided by metrics (and anyone who has a modicum of intuition, will always pick the metric that can be most gamed for himself, and his underlings). Also, if we're being real with ourselves, there's very little that managers "control," so picking a "real" metric would just breed an environment that selects for luck (or lies). In my opinion, the only useful metric is profit generated. Everything else is just an emotional safety blanket for uncertainty.

marginalia_nu 5y ago
> In my opinion, the only useful metric is profit generated. Everything else is just an emotional safety blanket for uncertainty. This too can be a toxic metric. Only focusing on what looks good in the quarterly report encourages a type of short-sightedness that has been the undoing of many businesses.
- zaidf 5y ago
  In my opinion, the only useful metric is profit generated We know by observing private equity that the easiest way to hack this metric is by doing layoffs until there is imminent threat of the business vanishing.
  - dsr_ 5y ago
    It's also not observable in real-time; often only quarterly or annually. That makes it really hard to make tactical decisions.
- throwaway2331 5y ago
  Yes; it's a trade-off due to how difficult it is to make people give a shit about the short-term---forget about the long-term (if there's no stock involved. We've barely figured out how to make CEOs tuned into the "long-term"). The corp is just a vehicle for the goals of all its passengers. Most of them are only on-board for a couple of stops, and then off to the next vehicle. To voluntarily forgo getting off at your stop, without receiving something of equal or greater value, seems awfully foolish.
cconstantine 5y ago
The only thing that matters is revenue, and if it can't be easily measured against revenue it doesn't exist. I am so tired of this line of reasoning. It's poisoning the industry, ruining products, and sucking the joy out of work. What about making a quality product because you have a passion for it? Nope, make a broken MVP and move on to the next customer. If I can't maintain a metric until I can tie it to revenue, then I guess I don't get cpu usage on our database. Or disk usage. Or response time for anything that only employees touch. Bad data and errors are fine as long as no one complains. Actually, if we produce a shitty product long enough everyone will get used to it and stop complaining. If we stop measuring error rates we can stop wasting time on fixing things right? Tests and code refactors just get in the way of delivering that MVP. Rewards are doled out for fixing an outage, and preventing them is a cost-center to be eliminated. On the flip side, now we've got KPIs for our ability to push things through the corporate bureaucracy. I didn't become a programmer so I can waste time estimating how long it'll take to produce a design doc that'll be useless by the time I get to implementation, just so a bureaucrat can decide if some deeply technical problem they don't understand should be fixed. There has to be a better way.

phillipcarter 5y ago

I don't have much to add other than this reminds me of the obsession around "satisfaction" for product metrics and why that seems to always boil down to slapping NPS on it because it's a number and execs like numbers. I think this is essential reading if anyone's interested on a product-focused tangent for the topic: https://articles.uie.com/net-promoter-score-considered-harmful-and-what-ux-professionals-can-do-about-it/ https://articles.uie.com/net-promoter-score-considered-harmf...

splittingTimes 5y ago
Sorry to ask but What's NPS?
- cconstantine 5y ago
  Net Promoter Score: https://en.wikipedia.org/wiki/Net_promoter_score https://en.wikipedia.org/wiki/Net_promoter_score How likely are people to promote the thing?
malshe 5y ago
I really liked this article. Thanks for sharing!

deleted 5y ago

[deleted]

a9h74j 5y ago

FWIW [from having skimmed] This should be of interest to people interested in UX where there are safety ramifications. Thus, the original article title might be a bit misleading. It is not necessarily about businesses running based upon illusions.

ngcc_hk 5y ago

“I'll barge in and say that there's this meme in business schools that: "what gets measured, gets done."” The key is those that sone might not be most useful.

tpoacher 5y ago

Goodhart's law is often confused with Campbell's law. The two are indeed very similar, and often used interchangeably, but there's an important difference in their phrasing. Goodhart's law states that: "When a measure becomes a target, it ceases to be a good measure." Campbell's law states that: "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." In other words, the former focuses on the fact that, while metrics sound good in theory, in practice they cannot be relied on; whereas the latter focuses on the fact that enforcing metrics in an attempt to control a social process tends to backfire. Or, in Trump language: Goodhart = "Sounds good, doesn't work" Campbell = "Worst trade deal ever"