5 ms·

As I read through the 50-odd comments, I wound up with a kanban / 5 whys level thought. All these points of discussion revolve around pluses and minuses of met

As I read through the 50-odd comments, I wound up with a kanban / 5 whys level thought. All these points of discussion revolve around pluses and minuses of metrics, but the next level is "Why establish metrics at all?" At a deeper (very generalized) level, we've been infected by production line, Deming style, efficiency focus. If you are handling previously defined routines, great. Optimizing can help you. As soon as you are working with adding value, be it in code, education, product development, metrics are all premature optimization. Promote the idea that you can't apply optimization techniques to creative steps and encourage managers to get off the numerical crutches and make value judgments. Takes a better manager but then that's the point anyway, right?

hopler 8y ago
Which businesses succeed without a business plan? Without metrics, how do you keep your runway open?
- pathseeker 8y ago
  People manage to survive without planning out the number of breaths they will take every day. A competent engineering team is going to do well even without target lines of code per second goals.
  - NyxWulf 8y ago
    I would argue that's counter to your intended point though. If you go into the E.R. or the hospital, one of the vitals they measure is in fact the number of breaths taken per unit of time. In a normally functioning human we don't have to worry about that metric because it is self regulating. If on the other hand you are in critical condition, measuring your breaths is a very important metric. So as I've said elsewhere, the problem isn't measurement. The problem is as Deming said "The aim defines the system". Well thought out measures in accordance with clear goals and aims can be valuable. Like anything though, it requires careful consideration and may need to be revised based on outcomes.
  - lazyasciiart 8y ago
    I see a lot of straw man arguments in this thread. There are bad metrics - I think that’s uncontroversial. But you (and others) appear to be making the argument that there are no good metrics - and pointing to bad metrics is irrelevant there.
- katbyte 8y ago
  When it comes to creative endeavors metrics rarely assist (ignoring code performance metrics like resident memory size or movie sales) You focus on your goal and try and take the best path there. LoC written or test coverages are distractions imho. Trust the skilled talented people you’ve hired as they are motivated to make the correct choices.
kuzehanka 8y ago
I can think of two deeper whys. 1: Dilution of responsibility. Metrics are defined by committee, which in turn answers to another strategic leadership committee. No single person is responsible for failure. Mediocre managers recite the entrenched KPI mantra because it cannot fail. 2: Differentiability. If you can't measure how well you're performing, you can't tell if you're improving or regressing with any given change. Total revenue or other coarse metrics are so far divorced from any given decision that they can't provide feedback about it. Just like gradient descent needs the underlying neural network to be differentiable, a business strategy needs the performance to be differentiable. KPIs are an attempt at that. Ultimately, KPIs suck but give a fairly predictable average outcome. Not doing KPIs is a risk that most leadership is unwilling to take. KPIs sit in the same category of concepts as airport theatre security. They're a product of (find something you can measure and try to manipulate that number, no matter how meaningless | something must be done. This solution is something. Something is better then nothing. Therefore let's do this).
- ergothus 8y ago
  > If you can't measure how well you're performing, There are a lot of assumptions baked into that statement, not least among them: (1) that being unable to put a number on it means you can't tell if you're improving or not; (2) that this all boils down to a number anyway (or a reasonable set of numbers); (3) That such numbers are actually predictive. There are two primary reasons people complain about metrics in my experience: (1) They're difficult/tedious to get and/or (2) they lead companies to focus on the wrong actions. These would belie the claims you're making. (I can't tell if you're making them personally, or just echoing what you've heard, so I'll argue against the generic theory) Turn your scenario around: We have a system. Experts in the field agree the system is more likely to cause poor decisions than not. Should we continue the system? (And trying to gather metrics on whether the system works or not is not the solution, because the question at hand is if metrics accurately guide decision-making.)
  - kuzehanka 8y ago
    You are preaching to the choir. I am just explaining why KPIs are prevalent and probably not going away any time soon. The business world is full of mediocre people who will cling to mediocre but predictable methodologies rather than take any kind of risk, especially if that risk is based in their own competence. The devil you know, etc.
superlopuh 8y ago
I'm not sure how to train machine learning models before defining metrics. I wonder how many other approaches/tools make metrics unavoidable. I think the key here is not marrying the metrics, and being happy to swap them out.
Jill_the_Pill 8y ago
"we've been infected by production line, Deming style, efficiency focus" Could you say more about this? I'm trying to form an opinion on this Deming-ism. My impression is that the actual Mr. Deming gave different advice: "Eliminate slogans, exhortations, and targets . . . Eliminate quotas, numbers, numerical goals."
- fsloth 8y ago
  I presume OP intended to write Taylorian?
  - C1sc0cat 8y ago
    Possibly but Taylorism is a term I have heard used
  - kome 8y ago
    https://en.wikipedia.org/wiki/Taylorism https://en.wikipedia.org/wiki/Taylorism
- pjc50 8y ago
  Yes, to me it seems that the real enemy is more Taylorism, especially in how it relates to advance planning.
  - NyxWulf 8y ago
    Generally I think the problem isn't in goals and measures, but in poor goals and measures. Deming said "The aim defines the system". So if you focus on short sighted metrics you will define a poor system. Metrics and reward systems need to be carefully considered and well thought out with regard to the larger scale system outcomes.
    - bunderbunder 8y ago
      That, in turn, tends to have a lot to do with the incentive structures of the person coming up with the metrics. If you've got someone who's been tasked with improving large-scale system outcomes, and they're developing metrics that are meant to be a means to that end, then you might expect them to come up with a well-crafted set of metrics. Of course, you can only expect them to implement those metrics to the extent that they're able to alter the system in a way that allows for collecting them. The best plan in the world isn't worth much if nobody has the power to execute on it. If, on the other hand, they're being designed by someone whose task was, "We need to be more data-driven; come up with some metrics and give me a dashboard," they're going to do exactly that. No more, no less.
bunderbunder 8y ago
If it were only premature optimization, it wouldn't be that bad. But it tends to lead to actively counterproductive optimization. Teams get incentivized to do things that increase their KPIs even when what they're doing delivers negative value. For example, I've definitely seen this happen on Agile teams where management makes KPIs based on velocity or ticket flow: Suddenly, lots and lots and lots of waste and technical debt are generated due to implementing unnecessary things. Because unquestioningly implementing tickets as fast as you can increases velocity, while spending time to stop and talk about whether or not something is a good idea, or can be solved in a better way, decreases it.

hopler 8y ago

Which businesses succeed without a business plan? Without metrics, how do you keep your runway open?

pathseeker 8y ago
People manage to survive without planning out the number of breaths they will take every day. A competent engineering team is going to do well even without target lines of code per second goals.
- NyxWulf 8y ago
  I would argue that's counter to your intended point though. If you go into the E.R. or the hospital, one of the vitals they measure is in fact the number of breaths taken per unit of time. In a normally functioning human we don't have to worry about that metric because it is self regulating. If on the other hand you are in critical condition, measuring your breaths is a very important metric. So as I've said elsewhere, the problem isn't measurement. The problem is as Deming said "The aim defines the system". Well thought out measures in accordance with clear goals and aims can be valuable. Like anything though, it requires careful consideration and may need to be revised based on outcomes.
- lazyasciiart 8y ago
  I see a lot of straw man arguments in this thread. There are bad metrics - I think that’s uncontroversial. But you (and others) appear to be making the argument that there are no good metrics - and pointing to bad metrics is irrelevant there.
katbyte 8y ago
When it comes to creative endeavors metrics rarely assist (ignoring code performance metrics like resident memory size or movie sales) You focus on your goal and try and take the best path there. LoC written or test coverages are distractions imho. Trust the skilled talented people you’ve hired as they are motivated to make the correct choices.

kuzehanka 8y ago

I can think of two deeper whys. 1: Dilution of responsibility. Metrics are defined by committee, which in turn answers to another strategic leadership committee. No single person is responsible for failure. Mediocre managers recite the entrenched KPI mantra because it cannot fail. 2: Differentiability. If you can't measure how well you're performing, you can't tell if you're improving or regressing with any given change. Total revenue or other coarse metrics are so far divorced from any given decision that they can't provide feedback about it. Just like gradient descent needs the underlying neural network to be differentiable, a business strategy needs the performance to be differentiable. KPIs are an attempt at that. Ultimately, KPIs suck but give a fairly predictable average outcome. Not doing KPIs is a risk that most leadership is unwilling to take. KPIs sit in the same category of concepts as airport theatre security. They're a product of (find something you can measure and try to manipulate that number, no matter how meaningless | something must be done. This solution is something. Something is better then nothing. Therefore let's do this).

ergothus 8y ago
> If you can't measure how well you're performing, There are a lot of assumptions baked into that statement, not least among them: (1) that being unable to put a number on it means you can't tell if you're improving or not; (2) that this all boils down to a number anyway (or a reasonable set of numbers); (3) That such numbers are actually predictive. There are two primary reasons people complain about metrics in my experience: (1) They're difficult/tedious to get and/or (2) they lead companies to focus on the wrong actions. These would belie the claims you're making. (I can't tell if you're making them personally, or just echoing what you've heard, so I'll argue against the generic theory) Turn your scenario around: We have a system. Experts in the field agree the system is more likely to cause poor decisions than not. Should we continue the system? (And trying to gather metrics on whether the system works or not is not the solution, because the question at hand is if metrics accurately guide decision-making.)
- kuzehanka 8y ago
  You are preaching to the choir. I am just explaining why KPIs are prevalent and probably not going away any time soon. The business world is full of mediocre people who will cling to mediocre but predictable methodologies rather than take any kind of risk, especially if that risk is based in their own competence. The devil you know, etc.

superlopuh 8y ago

I'm not sure how to train machine learning models before defining metrics. I wonder how many other approaches/tools make metrics unavoidable. I think the key here is not marrying the metrics, and being happy to swap them out.

Jill_the_Pill 8y ago

"we've been infected by production line, Deming style, efficiency focus" Could you say more about this? I'm trying to form an opinion on this Deming-ism. My impression is that the actual Mr. Deming gave different advice: "Eliminate slogans, exhortations, and targets . . . Eliminate quotas, numbers, numerical goals."

fsloth 8y ago
I presume OP intended to write Taylorian?
- C1sc0cat 8y ago
  Possibly but Taylorism is a term I have heard used
- kome 8y ago
  https://en.wikipedia.org/wiki/Taylorism https://en.wikipedia.org/wiki/Taylorism
pjc50 8y ago
Yes, to me it seems that the real enemy is more Taylorism, especially in how it relates to advance planning.
- NyxWulf 8y ago
  Generally I think the problem isn't in goals and measures, but in poor goals and measures. Deming said "The aim defines the system". So if you focus on short sighted metrics you will define a poor system. Metrics and reward systems need to be carefully considered and well thought out with regard to the larger scale system outcomes.
  - bunderbunder 8y ago
    That, in turn, tends to have a lot to do with the incentive structures of the person coming up with the metrics. If you've got someone who's been tasked with improving large-scale system outcomes, and they're developing metrics that are meant to be a means to that end, then you might expect them to come up with a well-crafted set of metrics. Of course, you can only expect them to implement those metrics to the extent that they're able to alter the system in a way that allows for collecting them. The best plan in the world isn't worth much if nobody has the power to execute on it. If, on the other hand, they're being designed by someone whose task was, "We need to be more data-driven; come up with some metrics and give me a dashboard," they're going to do exactly that. No more, no less.

bunderbunder 8y ago

If it were only premature optimization, it wouldn't be that bad. But it tends to lead to actively counterproductive optimization. Teams get incentivized to do things that increase their KPIs even when what they're doing delivers negative value. For example, I've definitely seen this happen on Agile teams where management makes KPIs based on velocity or ticket flow: Suddenly, lots and lots and lots of waste and technical debt are generated due to implementing unnecessary things. Because unquestioningly implementing tickets as fast as you can increases velocity, while spending time to stop and talk about whether or not something is a good idea, or can be solved in a better way, decreases it.