7 ms·

Metrics-driven product development is hard

74 points by kiyanwang 5y ago

coldcode 5y ago
At my last (very large but not FAANG) employer we collected huge amounts of metrics in our apps, ignored them entirely, and based everything on executive whims. Is this Whim-Driven Development?
- foobiekr 5y ago
  This is sometimes called "reality."
- kqr 5y ago
  Known as HiPPO -- highest-paid person's opinion.
ergest 5y ago
That’s because metrics are great for optimization and fine tuning features but terrible for innovation
- rch 5y ago
  One reason might be that great innovators don't naturally gravitate towards middle management (or management consulting), so the metrics are flawed out of the gate.
  - dwaltrip 5y ago
    We don't have the tools to measure the complex dynamics and factors that result in innovation.
- kqr 5y ago
  Do you have evidence for this claim? I think metrics are absolutely invaluable for innovation. What is innovation but throwing 100 things at the wall and seeing which two of them stick? And to quickly get a sense of which things have stuck to the wall, metrics are the only reliable way.
  - smugglerFlynn 5y ago
    It probably depends on the type of innovation. If you are trying to do things differently, metrics are invaluable — most modern internal innovation teams use metrics to build and manage an idea funnel. However, if your innovation strategy is to do different things, you will likely be focused on market differentiation, customer development etc. Common practice is to refer to AARRR metrics for this, but it’s not like metrics themselves are the main focus.
    - netjiro 5y ago
      Anecdata: on a couple of projects our metrics have told us that do things differently will not get us to target, forcing hard choices to go do different things to get there. But it is usually an uncomfortable fight to get the boat turned around, especially if people have already invested a lot of money and ego in the earlier decision tree.
      - smugglerFlynn 5y ago
        Also purely anecdotal, but I think it takes a different skillset to pivot correctly, which execs who work in a corporate environment focused on doing things differently often lack. That’s where external consultants usually get involved.
ford 5y ago
Metrics are great as an objective measure of feature releases/goals/success, but they don't provide the whole picture in terms of product success or direction. Years ago I interned on a messaging team at a FAANG company where everything was driven by ~3 metrics. We had a screen with the trailing 30d average of our metrics, future work was prioritized based on the estimated impact to the metric, and the success of our team was based on these 3 metrics. My intern project involved some machine learning modeling using the (anonymized/scrubbed) content of the messages sent with our product, and despite the team having existed for >12 months, this was the first time anyone on the team had read a message sent with the product. In this case, the sharp focus on metrics was super useful for prioritizing features and evaluating our success - but was done so to a fault when the team didn't have a concrete idea of what people were trying to do with our product. The usual solution to this is to spend more time talking to users - it'd be interesting if there was a process that can help teams know how much time to spend on metrics vs qualitative feedback
baybal2 5y ago
Metrics driven software development gets more meaningless, the bigger is the customer base, and frequent are releases. A good way to make a big organisation chase meaningless things without your bosses realising, and even get a promotion for it.
- nowherebeen 5y ago
  Can you give an example of this? I am having a difficult time understanding your meaning.
  - baybal2 5y ago
    Google Chrome is a great example "developers in the hamster wheel" of metrics. Keeps getting constantly redesigned for seemingly no reason, and certainly not all for the better.
    - Nextgrid 5y ago
      Google Chrome's (or any other established software's) useless changes are absolutely metric-driven. The metric is some employees' salaries.
beebmam 5y ago
Metrics-driven product development sounds like a good idea but in practice ends up being Cargo Culting. That's usually because people rarely reflect on the metrics they're deriving meaning from, rarely abandoning ambiguous metrics as misleading, and rarely imagining new metrics to find utility. Those are very hard tasks that take time and deep thought.
karaterobot 5y ago
The last company I worked at was very metrics-driven, and used a really methodical, numbers-driven approach to make big, unpopular, counter-intuitive bets that failed catastrophically, while ruining morale, and trust in leadership, and gutting the company. After that experience, my view of metrics-driven product development is that it's a way to offload the burden of thinking too hard or understanding where you're going. When people talk about metrics-driven product development, I think of drivers who blindly follow their GPS' directions into a lake. Obviously, metrics can be a useful supplement to a deep understanding of the product, the domain, the users, and the company, but keep it on a leash.
- akiselev 5y ago
  > After that experience, my view of metrics-driven product development is that it's a way to offload the burden of thinking too hard or understanding where you're going. When people talk about metrics-driven product development, I think of drivers who blindly follow their GPS' directions into a lake. There's also the flip side - metrics-driven development used to keep other people from thinking too hard and scrutinizing work. At my last company, many product decisions were justified using a litany of metrics including AB tests, which people just swallowed because "the data speaks for itself!" I once took a look at the app we were using to run the AB tests and not a single one had reached statistical significance. The company was basing its decisions on data considered garbage even by the AB app's naive p-value algorithm. I thought that after dozens of A/B tests at least one would be statistically significant just by dumb luck but no, not a single test had even the veneer of credibility.
  - kqr 5y ago
    But what you're describing is noise-driven development. It's not an actual metric value until you've somewhat ruled out noise as the explanation.
    - akiselev 5y ago
      I believe that's a distinction without a difference. Most scientists have a hard enough time understanding statistical significance in well controlled experiments and they are (ostensibly) trained for it. There's so much noise in business data that it's like reading tea leaves, except no one can prove it because by the time they gather enough data they realize that the management and/or market dynamics have introduced even more noise so everyone keeps chugging along, blissfully unaware that they're indulging in statistical fantasies and reinforcing whatever beliefs they already hold.
black_13 5y ago
tlarkworthy 5y ago
metrics driven development has killed facebook.com. On its own, it is too short sighted. How do you measure long term success? hint: u can't. Blind application of ML is therefore, also dangerous for similar reasons. metrics are useful for tightening the tactical decisions like, fuck, our landing page conversion is low so let's invest there, but strategically metrics are unable to be creative and innovative and all the other things you need for a great product.
dwb 5y ago
Metrics are great for testing a quantifiable hypothesis, but not all worthwhile hypotheses are quantifiable, and in ignoring the latter we discount an essentially human side to creativity and that’s a big turn-off for me.
deleted 5y ago
[deleted]
debarshri 5y ago
We have been using metrics driven product development lately. What we have been doing is hypothesize what will grow our product feature based on org identity, talking to customers, looking at the market. We define some metrics which we believe represents the success metrics in the best way possible. Some are based on domain expertise, some are just best guess. We use these metrics as a way to ascertain the hypothesis. We are not using it as indicator to build the next thing. The thing is that the success hypothesis or strategy might be wrong so we are willing to throw that away and start all over again. Not sure if it is the best way to do it.
indymike 5y ago
Metrics are really good, and really can help you reason about what will happen to usage when you add, remove or change a feature. There are two better / essential tools: First, dog food / use your product if you can. I build recruiting software. When I hire developers, I am a user, just like my customers. You'll really feel the impact of defects and bad assumptions have, and it will really help you understand how to make users happy. The second is to work closely with a few customers. This way you have someone who can heat check the metrics and confirm they are having the same experience you do when you dog food, to make sure you are not being led to an alternate reality.
jarjoura 5y ago
Metrics is insanely useful to help inform all kinds of product decisions. The most basic example I can think of, detecting when people aren't using a particular feature you've built. I think where it gets tricky and where even FAANG companies fail at, is the intersection of intuition and validation. Intuition can guide you along initially, ie, we need to build this thing because this it could become our differentiator and would be useful. Well then you need some way to validate that your intuition is correct. So how do you do that? At FAANG companies metrics can come from all kinds of sources, not just a few various MixPanel event streams. UX researchers can go out and do real world case studies and bring back statistics. User feedback itself can come through Marketing outreach or surveys. PMs collect research on similar products and add valuable data to goal against. All this to say, when you hear the word Metrics, do not assume it's just a single stream of usage data. Yet still, even when FAANG companies get all these data points from all over, they still make invalid decisions too. It's really hard for everyone! The point I'm making though, use Metrics to help inform, but also understand that Metrics should never become dogma. They will help give clarity in many many things, yes, even early indicators of things. However, they are never going to invent the next big thing. That will be up to your creative teams who work outside the metric bubble to push in different ways to see where things could go.

74 points by kiyanwang 5y ago

At my last (very large but not FAANG) employer we collected huge amounts of metrics in our apps, ignored them entirely, and based everything on executive whims. Is this Whim-Driven Development?

foobiekr 5y ago
This is sometimes called "reality."
kqr 5y ago
Known as HiPPO -- highest-paid person's opinion.

ergest 5y ago

That’s because metrics are great for optimization and fine tuning features but terrible for innovation

rch 5y ago
One reason might be that great innovators don't naturally gravitate towards middle management (or management consulting), so the metrics are flawed out of the gate.
- dwaltrip 5y ago
  We don't have the tools to measure the complex dynamics and factors that result in innovation.
kqr 5y ago
Do you have evidence for this claim? I think metrics are absolutely invaluable for innovation. What is innovation but throwing 100 things at the wall and seeing which two of them stick? And to quickly get a sense of which things have stuck to the wall, metrics are the only reliable way.
- smugglerFlynn 5y ago
  It probably depends on the type of innovation. If you are trying to do things differently, metrics are invaluable — most modern internal innovation teams use metrics to build and manage an idea funnel. However, if your innovation strategy is to do different things, you will likely be focused on market differentiation, customer development etc. Common practice is to refer to AARRR metrics for this, but it’s not like metrics themselves are the main focus.
  - netjiro 5y ago
    Anecdata: on a couple of projects our metrics have told us that do things differently will not get us to target, forcing hard choices to go do different things to get there. But it is usually an uncomfortable fight to get the boat turned around, especially if people have already invested a lot of money and ego in the earlier decision tree.
    - smugglerFlynn 5y ago
      Also purely anecdotal, but I think it takes a different skillset to pivot correctly, which execs who work in a corporate environment focused on doing things differently often lack. That’s where external consultants usually get involved.

ford 5y ago

Metrics are great as an objective measure of feature releases/goals/success, but they don't provide the whole picture in terms of product success or direction. Years ago I interned on a messaging team at a FAANG company where everything was driven by ~3 metrics. We had a screen with the trailing 30d average of our metrics, future work was prioritized based on the estimated impact to the metric, and the success of our team was based on these 3 metrics. My intern project involved some machine learning modeling using the (anonymized/scrubbed) content of the messages sent with our product, and despite the team having existed for >12 months, this was the first time anyone on the team had read a message sent with the product. In this case, the sharp focus on metrics was super useful for prioritizing features and evaluating our success - but was done so to a fault when the team didn't have a concrete idea of what people were trying to do with our product. The usual solution to this is to spend more time talking to users - it'd be interesting if there was a process that can help teams know how much time to spend on metrics vs qualitative feedback

baybal2 5y ago

Metrics driven software development gets more meaningless, the bigger is the customer base, and frequent are releases. A good way to make a big organisation chase meaningless things without your bosses realising, and even get a promotion for it.

nowherebeen 5y ago
Can you give an example of this? I am having a difficult time understanding your meaning.
- baybal2 5y ago
  Google Chrome is a great example "developers in the hamster wheel" of metrics. Keeps getting constantly redesigned for seemingly no reason, and certainly not all for the better.
  - Nextgrid 5y ago
    Google Chrome's (or any other established software's) useless changes are absolutely metric-driven. The metric is some employees' salaries.

beebmam 5y ago

Metrics-driven product development sounds like a good idea but in practice ends up being Cargo Culting. That's usually because people rarely reflect on the metrics they're deriving meaning from, rarely abandoning ambiguous metrics as misleading, and rarely imagining new metrics to find utility. Those are very hard tasks that take time and deep thought.

karaterobot 5y ago

The last company I worked at was very metrics-driven, and used a really methodical, numbers-driven approach to make big, unpopular, counter-intuitive bets that failed catastrophically, while ruining morale, and trust in leadership, and gutting the company. After that experience, my view of metrics-driven product development is that it's a way to offload the burden of thinking too hard or understanding where you're going. When people talk about metrics-driven product development, I think of drivers who blindly follow their GPS' directions into a lake. Obviously, metrics can be a useful supplement to a deep understanding of the product, the domain, the users, and the company, but keep it on a leash.

akiselev 5y ago
> After that experience, my view of metrics-driven product development is that it's a way to offload the burden of thinking too hard or understanding where you're going. When people talk about metrics-driven product development, I think of drivers who blindly follow their GPS' directions into a lake. There's also the flip side - metrics-driven development used to keep other people from thinking too hard and scrutinizing work. At my last company, many product decisions were justified using a litany of metrics including AB tests, which people just swallowed because "the data speaks for itself!" I once took a look at the app we were using to run the AB tests and not a single one had reached statistical significance. The company was basing its decisions on data considered garbage even by the AB app's naive p-value algorithm. I thought that after dozens of A/B tests at least one would be statistically significant just by dumb luck but no, not a single test had even the veneer of credibility.
- kqr 5y ago
  But what you're describing is noise-driven development. It's not an actual metric value until you've somewhat ruled out noise as the explanation.
  - akiselev 5y ago
    I believe that's a distinction without a difference. Most scientists have a hard enough time understanding statistical significance in well controlled experiments and they are (ostensibly) trained for it. There's so much noise in business data that it's like reading tea leaves, except no one can prove it because by the time they gather enough data they realize that the management and/or market dynamics have introduced even more noise so everyone keeps chugging along, blissfully unaware that they're indulging in statistical fantasies and reinforcing whatever beliefs they already hold.

black_13 5y ago

tlarkworthy 5y ago

metrics driven development has killed facebook.com. On its own, it is too short sighted. How do you measure long term success? hint: u can't. Blind application of ML is therefore, also dangerous for similar reasons. metrics are useful for tightening the tactical decisions like, fuck, our landing page conversion is low so let's invest there, but strategically metrics are unable to be creative and innovative and all the other things you need for a great product.

dwb 5y ago

Metrics are great for testing a quantifiable hypothesis, but not all worthwhile hypotheses are quantifiable, and in ignoring the latter we discount an essentially human side to creativity and that’s a big turn-off for me.

deleted 5y ago

[deleted]

debarshri 5y ago

We have been using metrics driven product development lately. What we have been doing is hypothesize what will grow our product feature based on org identity, talking to customers, looking at the market. We define some metrics which we believe represents the success metrics in the best way possible. Some are based on domain expertise, some are just best guess. We use these metrics as a way to ascertain the hypothesis. We are not using it as indicator to build the next thing. The thing is that the success hypothesis or strategy might be wrong so we are willing to throw that away and start all over again. Not sure if it is the best way to do it.

indymike 5y ago

Metrics are really good, and really can help you reason about what will happen to usage when you add, remove or change a feature. There are two better / essential tools: First, dog food / use your product if you can. I build recruiting software. When I hire developers, I am a user, just like my customers. You'll really feel the impact of defects and bad assumptions have, and it will really help you understand how to make users happy. The second is to work closely with a few customers. This way you have someone who can heat check the metrics and confirm they are having the same experience you do when you dog food, to make sure you are not being led to an alternate reality.

jarjoura 5y ago

Metrics is insanely useful to help inform all kinds of product decisions. The most basic example I can think of, detecting when people aren't using a particular feature you've built. I think where it gets tricky and where even FAANG companies fail at, is the intersection of intuition and validation. Intuition can guide you along initially, ie, we need to build this thing because this it could become our differentiator and would be useful. Well then you need some way to validate that your intuition is correct. So how do you do that? At FAANG companies metrics can come from all kinds of sources, not just a few various MixPanel event streams. UX researchers can go out and do real world case studies and bring back statistics. User feedback itself can come through Marketing outreach or surveys. PMs collect research on similar products and add valuable data to goal against. All this to say, when you hear the word Metrics, do not assume it's just a single stream of usage data. Yet still, even when FAANG companies get all these data points from all over, they still make invalid decisions too. It's really hard for everyone! The point I'm making though, use Metrics to help inform, but also understand that Metrics should never become dogma. They will help give clarity in many many things, yes, even early indicators of things. However, they are never going to invent the next big thing. That will be up to your creative teams who work outside the metric bubble to push in different ways to see where things could go.