8 ms·

Don’t Let Metrics Undermine Your Business

103 points by prostoalex 7y ago

cripti7 7y ago
It seems this discussion is many years old. Of course, metrics won't help you run the business better, if you don't know how to run the business. But if you do, you can't do it without metrics. Nothing can be executed, if you can't measure it. Read this blog post, it's relevant: https://www.yegor256.com/2018/10/09/can-you-control-us.html https://www.yegor256.com/2018/10/09/can-you-control-us.html
6gvONxR4sf7o 7y ago
I work as a data scientist on an extremely data driven team at an extremely data driven company, and I couldn't agree more strongly. Everyone I work with would agree too. "Numbers aren't everything!" says the person whose job it is to deal with business problems via numbers. But this advice is so hard to implement. It's like the whole null hypothesis p-value debacle going on in academic publishing right now. Sure we can agree it's no good, but there's no standard alternative. Standards are powerful. They break lots of ties. All else equal, if you have a hunch and I have a measurement tied to objective reality, people will side with me. Nobody ever got fired for choosing Java. Nobody ever got fired for doing the thing that made The Metrics go up. It's so reassuring to just say "We still disagree, so let's just test it." We're submitting our dispute to adjudication via Science by resolving via The Metrics! It's much harder to be objective without these kinds of numbers, and I like being objective! I want the scientific method to tell me I did the right thing and I'm a good person. Because without that, when we're trying to figure out what to do, it's just us, you know? It feels like guessing, and that's hard.
- jbob2000 7y ago
  This is a recipe for mediocrity. Great things are created by people following their gut, not their metrics. Jobs didn’t have a data scientist telling him to make the iPhone. A data scientist would have told Notch to make a Call of Duty clone instead of Minecraft. There was no data scientist behind Gates, Warren, Zuckerberg, Musk, or anyone of their ilk. Looking back in history, some great discoveries and inventions were held back because the “data” said it wasn’t possible. Everybody thought ulcers were caused by stress and bad food, then Barry Marshall came along and literally trusted his gut by eating a bunch of bacteria to prove that’s how ulcers were formed. At best, data science helps you squeeze a couple extra bucks from someone else’s ingenuity. Once Jobs died, the data scientists got their hands on the iPhone and chopped it up into different models. And it was the data scientists who made your Facebook feed ad-ridden junk. And at worst, data science is destroying our society. It’s the data scientists who are driving this insane data vacuum that is plaguing the internet and our daily lives. You thwarted the 2016 election by weaponizing data against America. You did it again with brexit. No doubt you will be out in full force for every major political event for the next few decades, sullying the discourse of the electorate. Bit of a rant... sorry. I hate this over-reliance on data and the smugness it’s captors carry.
  - mdorazio 7y ago
    I would say it's a recipe for continuation rather than innovation. You're unlikely to invent anything revolutionary just by looking at metrics, but you're also unlikely to keep a business going without them. It's easy to look at something like the iPhone and say, "metrics didn't invent that!" but metrics absolutely put millions of phones in the hands of customers, built an effective production and supply chain, optimized the whole experience, and turned Apple into a trillion dollar company. As with most things, there's a balance and a place for each.
    - jbob2000 7y ago
      I disagree that metrics helped Apple scale, but we’d have to wade in to a semantic debate to settle this. When you have a killer idea, scaling is obvious; just do more. More people, more machines, more materials, etc. You don’t need a number to tell you how much more to scale, because you always want the max anyways. Tim Cook was told there weren’t enough CNC machines in the world to manufacture the unibody MacBook. So he bought the company that makes CNC machines. What’s the point of metrics when you can shift the entire world to meet your needs? Great things are like that. Metrics are for mediocre products and services. We use them in banking because people fucking hate the banking system and we have to pull teeth to get customers to spend money. McDonald’s uses metrics to gauge how shitty they can make their beef without people noticing. Hollywood uses metrics to decide which movie franchise to reboot or make a sequel of. Metrics are a sign that you are done innovating and you are squeezing the sponge before someone else’s innovation dethrones yours.
      - nitwit005 7y ago
        > You don’t need a number to tell you how much more to scale, because you always want the max anyways. And you bankrupted the company building factories you didn't need, because iPhone demand has its limits.
      - leeoniya 7y ago
        i mostly agree with you. great intuition and experience can largely reduce the need for metrics. the problem is that not every business decision is clear cut and people who have to make them are not always Jobs or Gates. what metrics give you is a way to give the average joe (which is most of us) a way to take calculated risks when there are multiple stakeholders each with their own opinion of what should be done. metrics - like statistics - can be used effectively as a guide or can be used obsessively like dogma.
- hef19898 7y ago
  For me that depends on the level you are looking at: Tactical, Operational, Strategic. The first two have to be data and metrics driven, that's where you make or loose money right now. On the strategic level your performance metrics are giving only a baseline for your capabilities, strength and weaknesses. That should only be affecting the implementation of a given strategy and never the definition of it. And as always, if a metrics system is ill defined from the start it will steer you in the wrong direction. Like a broken speedometer in a car. And we didn't even start to talk about the corresponding incentive and bonus program yet...
  - heymijo 7y ago
    Schlitz beer is a great example of the downfall of managing metrics from a strategic level. They began cutting cost/quality of ingredients in their beer to compete with Anheuser Busch. "Salami slicing the company to death."[0] >* the steps from A to B and from B to C might have been tiny and unnoticeable, but the steps from A to M added up to a big leap.* [1] The metrics looked great, getting more beer out of less ingredients, in less time. Until the 70's when they hit a tipping point and drinkers stopped buying, tanking the company. [0] https://en.wikipedia.org/wiki/Salami_slicing https://en.wikipedia.org/wiki/Salami_slicing [1] https://beerconnoisseur.com/articles/how-milwaukees-famous-beer-became-infamous https://beerconnoisseur.com/articles/how-milwaukees-famous-b...
- heisenbit 7y ago
  Nobody has ever been fired for believing data.
  - username90 7y ago
    Unless it is data about political stuff like religion, race or gender. I bet someone has even gotten fired for believing in evolution.
    - concordDance 7y ago
      Bringing up Culture War ideas like this in unrelated topics is a good way to derail them
- plaidfuji 7y ago
  This speaks to a concept in data science I like to call "measurability bias" - the entire field of data science is biased toward making decisions on things that are, well, measurable. If you try to apply data science to a problem whose objective function doesn't have robust representative measurements tied to it, you are bound to fail. Being able to read subtle facial cues, measure the energy in a room, basically all of the subtle empathic interactions required to gauge customer satisfaction... we're not there yet.
- tryitnow 7y ago
  I think the whole "data-driven" ideology is getting worn out. It's especially bad on a lot of data science teams because there can be a lack of serious management experience and know-how if the team has a background mostly in academia. Academia is full of extremely dysfunctional behavior, even moreso than corporations, which is shockingly hard to pull off! Everything described in this article is old hat for experienced, knowledgeable executives, people in the know have been talking about these ideas for at least half a century (dating back to Drucker, Deming and others). While I agree with the authors' recommendations, I disagree about their diagnosis. I think Wells Fargo's main problem was a lack of internal controls, which is a boring, but necessary part of any large business.
olau 7y ago
It's interesting to ponder how a metric, through its design, is a psychological brain hack that trumps rational thought and common sense. Perhaps it's the gamification aspect that gets us? An example I think everyone can relate to is in education where grading is a rich source of perverse incentives. In developing minds, no less. Yet the metric persists. (Just to be clear, the alternative to grading is not no feedback, but personalized, situation-specific feedback. Freeing yourself from a bad metric can open doors.)
tomrandle 7y ago
Seems to me Surrogation is very similar to The Cobra Effect / Perverse Incentives (https://www.geckoboard.com/learn/data-literacy/statistical-fallacies/cobra-effect/ https://www.geckoboard.com/learn/data-literacy/statistical-f...) , or the McNamara fallacy: https://www.geckoboard.com/learn/data-literacy/statistical-fallacies/mcnamara-fallacy/ https://www.geckoboard.com/learn/data-literacy/statistical-f... Both very real problems that have existed forever. The solution, as the article suggests, is not relying on just a single metric, and not losing sight of your wider strategy. There’s a really good article on Reforge about the problems with the recent trend of a single “North Star Metric”: https://www.reforge.com/blog/north-star-metric-growth https://www.reforge.com/blog/north-star-metric-growth. Stacey Barr’s another thought leader on the subject: https://www.staceybarr.com/measure-up/a-single-kpi-that-measures-overall-business-performance/ https://www.staceybarr.com/measure-up/a-single-kpi-that-meas.... Metrics absolutely cannot replace strategy and vision. It needs to be a core value of the company that you’d rather do the right thing by the wider mission than drive a metric at any cost. Managers at every level need to live up to that. This is harder the bigger the company. It’s one of the things OKRs, love or hate them, try and help with. Like with everything it’s whether you comply with the spirit or the letter of the law. And whether people get rewarded for latter!
ThomPete 7y ago
Metrics are great for optimizing your business not for deciding which one to build.
- TeMPOraL 7y ago
  The problem talked about in the article is that it's not easy to express your optimization goals in terms of concrete metrics, and your optimization process will always optimize those metrics, not your real goals.
  - ThomPete 7y ago
    Yes, the art is always to pick the right metrics. The problem is when companies choose qualitative and thus interpretational metrics. Just stay to the simple easily trackable and quantitative ones and the make it your job to interpret them qualitatively.
    - TeMPOraL 7y ago
      Most real-life goals can't be easily expressed with few "easily trackable" metrics. Take example from the article, customer satisfaction. That's half of the problem; the other half is that metrics can be gamed, and they'll end up being gamed accidentally or for profit if you're not careful. Like, the easiest way to make a system stable is to make it so painful to use that people don't use it - and if they don't use it, they can't break it. Or, the example of return tracking from the TFA. I read a lot about how data-driven companies measure this and that, often through questionable, privacy-violating measures. What I don't read about is how do these companies ensure the metrics are actually valid - that they're correctly sampling the population[0], or that they're measuring what the authors think they're measuring, or that they're not being misreported or otherwise gamed (very common if the value of a metric impacts someone's career or even workload). -- [0] - e.g. voluntary surveys usually don't, telemetry increasingly doesn't either as more and more people are aware of it and disable it.
      - ThomPete 7y ago
666lumberjack 7y ago
This reminds me of Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." It seems like in the case of employees at a company, just knowing that a parameter is being measured can result in it implicitly becoming a target even when there is no explicit focus on it (especially if it is the only or one of a small number of metrics that is discussed). I have observed that in general most humans are very bad at considering the incentives created by rules and rewards they impose or seek to impose.
- dragonwriter 7y ago
  Goodhart’s Law really only applies to measures that aren't measuring the actual figure of merit but a proxy that is incorrectly assumed to be inextricably tied to it. This is very common, though, and a real problem associated with it is people pick proxies without clearly documenting (or even thinking through) what they want to measure, why they pick a particular proxy, and how the proxy might fall. As a result, people remote from the decision (including the original decision maker) tend to incautiously apply the proxy because they aren't even aware that it is not the actual figure of merit.
  - mikeash 7y ago
    The point of the law is that virtually every measure is a proxy to some extent. Finding a measure that exactly matches what you really want to optimize is almost impossible.
tryitnow 7y ago
This article represents one of the problems I have with HBR and "business school" literature in general. It's a rehash of tried and true principles that have been described numerous times before almost always without attribution to original sources. What's wrong with that? Well, for one the authors are passing off work that is clearly not their own. Fine, whatever, it's a b-school publication, nobody takes their intellectual output seriously anyways. But the more important problem is that you lose a sense of where this knowledge has come from, where it's been applied, how it's been applied, and other important context. In other words, reading this article is basically a waste of time. It's better to stick to a handful of classic business texts (Drucker's a good place to start and yes, he goes over all the recommendations these authors make in Effective Executive). Reading Kaplan on the Balanced Scorecard is another good place to start.
- heymijo 7y ago
  > But the more important problem is that you lose a sense of where this knowledge has come from, where it's been applied, how it's been applied, and other important context. Thank you for saying this! A thought I have had in the past few years is "there's been nothing new in management since Drucker and Deming." I go back and forth because some people have done a nice job of updating their ideas/applying them to modern disciplines, which is valuable. However, what you said about where that knowledge has come from/been applied/how/context is sorely missing. That these ideas are still not common knowledge leads to so many unnecessary mistakes.

103 points by prostoalex 7y ago

It seems this discussion is many years old. Of course, metrics won't help you run the business better, if you don't know how to run the business. But if you do, you can't do it without metrics. Nothing can be executed, if you can't measure it. Read this blog post, it's relevant: https://www.yegor256.com/2018/10/09/can-you-control-us.html https://www.yegor256.com/2018/10/09/can-you-control-us.html

6gvONxR4sf7o 7y ago

I work as a data scientist on an extremely data driven team at an extremely data driven company, and I couldn't agree more strongly. Everyone I work with would agree too. "Numbers aren't everything!" says the person whose job it is to deal with business problems via numbers. But this advice is so hard to implement. It's like the whole null hypothesis p-value debacle going on in academic publishing right now. Sure we can agree it's no good, but there's no standard alternative. Standards are powerful. They break lots of ties. All else equal, if you have a hunch and I have a measurement tied to objective reality, people will side with me. Nobody ever got fired for choosing Java. Nobody ever got fired for doing the thing that made The Metrics go up. It's so reassuring to just say "We still disagree, so let's just test it." We're submitting our dispute to adjudication via Science by resolving via The Metrics! It's much harder to be objective without these kinds of numbers, and I like being objective! I want the scientific method to tell me I did the right thing and I'm a good person. Because without that, when we're trying to figure out what to do, it's just us, you know? It feels like guessing, and that's hard.

jbob2000 7y ago
This is a recipe for mediocrity. Great things are created by people following their gut, not their metrics. Jobs didn’t have a data scientist telling him to make the iPhone. A data scientist would have told Notch to make a Call of Duty clone instead of Minecraft. There was no data scientist behind Gates, Warren, Zuckerberg, Musk, or anyone of their ilk. Looking back in history, some great discoveries and inventions were held back because the “data” said it wasn’t possible. Everybody thought ulcers were caused by stress and bad food, then Barry Marshall came along and literally trusted his gut by eating a bunch of bacteria to prove that’s how ulcers were formed. At best, data science helps you squeeze a couple extra bucks from someone else’s ingenuity. Once Jobs died, the data scientists got their hands on the iPhone and chopped it up into different models. And it was the data scientists who made your Facebook feed ad-ridden junk. And at worst, data science is destroying our society. It’s the data scientists who are driving this insane data vacuum that is plaguing the internet and our daily lives. You thwarted the 2016 election by weaponizing data against America. You did it again with brexit. No doubt you will be out in full force for every major political event for the next few decades, sullying the discourse of the electorate. Bit of a rant... sorry. I hate this over-reliance on data and the smugness it’s captors carry.
- mdorazio 7y ago
  I would say it's a recipe for continuation rather than innovation. You're unlikely to invent anything revolutionary just by looking at metrics, but you're also unlikely to keep a business going without them. It's easy to look at something like the iPhone and say, "metrics didn't invent that!" but metrics absolutely put millions of phones in the hands of customers, built an effective production and supply chain, optimized the whole experience, and turned Apple into a trillion dollar company. As with most things, there's a balance and a place for each.
  - jbob2000 7y ago
    I disagree that metrics helped Apple scale, but we’d have to wade in to a semantic debate to settle this. When you have a killer idea, scaling is obvious; just do more. More people, more machines, more materials, etc. You don’t need a number to tell you how much more to scale, because you always want the max anyways. Tim Cook was told there weren’t enough CNC machines in the world to manufacture the unibody MacBook. So he bought the company that makes CNC machines. What’s the point of metrics when you can shift the entire world to meet your needs? Great things are like that. Metrics are for mediocre products and services. We use them in banking because people fucking hate the banking system and we have to pull teeth to get customers to spend money. McDonald’s uses metrics to gauge how shitty they can make their beef without people noticing. Hollywood uses metrics to decide which movie franchise to reboot or make a sequel of. Metrics are a sign that you are done innovating and you are squeezing the sponge before someone else’s innovation dethrones yours.
    - nitwit005 7y ago
      > You don’t need a number to tell you how much more to scale, because you always want the max anyways. And you bankrupted the company building factories you didn't need, because iPhone demand has its limits.
    - leeoniya 7y ago
      i mostly agree with you. great intuition and experience can largely reduce the need for metrics. the problem is that not every business decision is clear cut and people who have to make them are not always Jobs or Gates. what metrics give you is a way to give the average joe (which is most of us) a way to take calculated risks when there are multiple stakeholders each with their own opinion of what should be done. metrics - like statistics - can be used effectively as a guide or can be used obsessively like dogma.
hef19898 7y ago
For me that depends on the level you are looking at: Tactical, Operational, Strategic. The first two have to be data and metrics driven, that's where you make or loose money right now. On the strategic level your performance metrics are giving only a baseline for your capabilities, strength and weaknesses. That should only be affecting the implementation of a given strategy and never the definition of it. And as always, if a metrics system is ill defined from the start it will steer you in the wrong direction. Like a broken speedometer in a car. And we didn't even start to talk about the corresponding incentive and bonus program yet...
- heymijo 7y ago
  Schlitz beer is a great example of the downfall of managing metrics from a strategic level. They began cutting cost/quality of ingredients in their beer to compete with Anheuser Busch. "Salami slicing the company to death."[0] >* the steps from A to B and from B to C might have been tiny and unnoticeable, but the steps from A to M added up to a big leap.* [1] The metrics looked great, getting more beer out of less ingredients, in less time. Until the 70's when they hit a tipping point and drinkers stopped buying, tanking the company. [0] https://en.wikipedia.org/wiki/Salami_slicing https://en.wikipedia.org/wiki/Salami_slicing [1] https://beerconnoisseur.com/articles/how-milwaukees-famous-beer-became-infamous https://beerconnoisseur.com/articles/how-milwaukees-famous-b...
heisenbit 7y ago
Nobody has ever been fired for believing data.
- username90 7y ago
  Unless it is data about political stuff like religion, race or gender. I bet someone has even gotten fired for believing in evolution.
  - concordDance 7y ago
    Bringing up Culture War ideas like this in unrelated topics is a good way to derail them
plaidfuji 7y ago
This speaks to a concept in data science I like to call "measurability bias" - the entire field of data science is biased toward making decisions on things that are, well, measurable. If you try to apply data science to a problem whose objective function doesn't have robust representative measurements tied to it, you are bound to fail. Being able to read subtle facial cues, measure the energy in a room, basically all of the subtle empathic interactions required to gauge customer satisfaction... we're not there yet.
tryitnow 7y ago
I think the whole "data-driven" ideology is getting worn out. It's especially bad on a lot of data science teams because there can be a lack of serious management experience and know-how if the team has a background mostly in academia. Academia is full of extremely dysfunctional behavior, even moreso than corporations, which is shockingly hard to pull off! Everything described in this article is old hat for experienced, knowledgeable executives, people in the know have been talking about these ideas for at least half a century (dating back to Drucker, Deming and others). While I agree with the authors' recommendations, I disagree about their diagnosis. I think Wells Fargo's main problem was a lack of internal controls, which is a boring, but necessary part of any large business.

olau 7y ago

It's interesting to ponder how a metric, through its design, is a psychological brain hack that trumps rational thought and common sense. Perhaps it's the gamification aspect that gets us? An example I think everyone can relate to is in education where grading is a rich source of perverse incentives. In developing minds, no less. Yet the metric persists. (Just to be clear, the alternative to grading is not no feedback, but personalized, situation-specific feedback. Freeing yourself from a bad metric can open doors.)

tomrandle 7y ago

Seems to me Surrogation is very similar to The Cobra Effect / Perverse Incentives (https://www.geckoboard.com/learn/data-literacy/statistical-fallacies/cobra-effect/ https://www.geckoboard.com/learn/data-literacy/statistical-f...) , or the McNamara fallacy: https://www.geckoboard.com/learn/data-literacy/statistical-fallacies/mcnamara-fallacy/ https://www.geckoboard.com/learn/data-literacy/statistical-f... Both very real problems that have existed forever. The solution, as the article suggests, is not relying on just a single metric, and not losing sight of your wider strategy. There’s a really good article on Reforge about the problems with the recent trend of a single “North Star Metric”: https://www.reforge.com/blog/north-star-metric-growth https://www.reforge.com/blog/north-star-metric-growth. Stacey Barr’s another thought leader on the subject: https://www.staceybarr.com/measure-up/a-single-kpi-that-measures-overall-business-performance/ https://www.staceybarr.com/measure-up/a-single-kpi-that-meas.... Metrics absolutely cannot replace strategy and vision. It needs to be a core value of the company that you’d rather do the right thing by the wider mission than drive a metric at any cost. Managers at every level need to live up to that. This is harder the bigger the company. It’s one of the things OKRs, love or hate them, try and help with. Like with everything it’s whether you comply with the spirit or the letter of the law. And whether people get rewarded for latter!

ThomPete 7y ago

Metrics are great for optimizing your business not for deciding which one to build.

TeMPOraL 7y ago
The problem talked about in the article is that it's not easy to express your optimization goals in terms of concrete metrics, and your optimization process will always optimize those metrics, not your real goals.
- ThomPete 7y ago
  Yes, the art is always to pick the right metrics. The problem is when companies choose qualitative and thus interpretational metrics. Just stay to the simple easily trackable and quantitative ones and the make it your job to interpret them qualitatively.
  - TeMPOraL 7y ago
    Most real-life goals can't be easily expressed with few "easily trackable" metrics. Take example from the article, customer satisfaction. That's half of the problem; the other half is that metrics can be gamed, and they'll end up being gamed accidentally or for profit if you're not careful. Like, the easiest way to make a system stable is to make it so painful to use that people don't use it - and if they don't use it, they can't break it. Or, the example of return tracking from the TFA. I read a lot about how data-driven companies measure this and that, often through questionable, privacy-violating measures. What I don't read about is how do these companies ensure the metrics are actually valid - that they're correctly sampling the population[0], or that they're measuring what the authors think they're measuring, or that they're not being misreported or otherwise gamed (very common if the value of a metric impacts someone's career or even workload). -- [0] - e.g. voluntary surveys usually don't, telemetry increasingly doesn't either as more and more people are aware of it and disable it.
    - ThomPete 7y ago

666lumberjack 7y ago

This reminds me of Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." It seems like in the case of employees at a company, just knowing that a parameter is being measured can result in it implicitly becoming a target even when there is no explicit focus on it (especially if it is the only or one of a small number of metrics that is discussed). I have observed that in general most humans are very bad at considering the incentives created by rules and rewards they impose or seek to impose.

dragonwriter 7y ago
Goodhart’s Law really only applies to measures that aren't measuring the actual figure of merit but a proxy that is incorrectly assumed to be inextricably tied to it. This is very common, though, and a real problem associated with it is people pick proxies without clearly documenting (or even thinking through) what they want to measure, why they pick a particular proxy, and how the proxy might fall. As a result, people remote from the decision (including the original decision maker) tend to incautiously apply the proxy because they aren't even aware that it is not the actual figure of merit.
- mikeash 7y ago
  The point of the law is that virtually every measure is a proxy to some extent. Finding a measure that exactly matches what you really want to optimize is almost impossible.

tryitnow 7y ago

This article represents one of the problems I have with HBR and "business school" literature in general. It's a rehash of tried and true principles that have been described numerous times before almost always without attribution to original sources. What's wrong with that? Well, for one the authors are passing off work that is clearly not their own. Fine, whatever, it's a b-school publication, nobody takes their intellectual output seriously anyways. But the more important problem is that you lose a sense of where this knowledge has come from, where it's been applied, how it's been applied, and other important context. In other words, reading this article is basically a waste of time. It's better to stick to a handful of classic business texts (Drucker's a good place to start and yes, he goes over all the recommendations these authors make in Effective Executive). Reading Kaplan on the Balanced Scorecard is another good place to start.

heymijo 7y ago
> But the more important problem is that you lose a sense of where this knowledge has come from, where it's been applied, how it's been applied, and other important context. Thank you for saying this! A thought I have had in the past few years is "there's been nothing new in management since Drucker and Deming." I go back and forth because some people have done a nice job of updating their ideas/applying them to modern disciplines, which is valuable. However, what you said about where that knowledge has come from/been applied/how/context is sorely missing. That these ideas are still not common knowledge leads to so many unnecessary mistakes.