4 ms·

The last company I worked at was very metrics-driven, and used a really methodical, numbers-driven approach to make big, unpopular, counter-intuitive bets that

The last company I worked at was very metrics-driven, and used a really methodical, numbers-driven approach to make big, unpopular, counter-intuitive bets that failed catastrophically, while ruining morale, and trust in leadership, and gutting the company. After that experience, my view of metrics-driven product development is that it's a way to offload the burden of thinking too hard or understanding where you're going. When people talk about metrics-driven product development, I think of drivers who blindly follow their GPS' directions into a lake. Obviously, metrics can be a useful supplement to a deep understanding of the product, the domain, the users, and the company, but keep it on a leash.

akiselev 5y ago
> After that experience, my view of metrics-driven product development is that it's a way to offload the burden of thinking too hard or understanding where you're going. When people talk about metrics-driven product development, I think of drivers who blindly follow their GPS' directions into a lake. There's also the flip side - metrics-driven development used to keep other people from thinking too hard and scrutinizing work. At my last company, many product decisions were justified using a litany of metrics including AB tests, which people just swallowed because "the data speaks for itself!" I once took a look at the app we were using to run the AB tests and not a single one had reached statistical significance. The company was basing its decisions on data considered garbage even by the AB app's naive p-value algorithm. I thought that after dozens of A/B tests at least one would be statistically significant just by dumb luck but no, not a single test had even the veneer of credibility.
- kqr 5y ago
  But what you're describing is noise-driven development. It's not an actual metric value until you've somewhat ruled out noise as the explanation.
  - akiselev 5y ago
    I believe that's a distinction without a difference. Most scientists have a hard enough time understanding statistical significance in well controlled experiments and they are (ostensibly) trained for it. There's so much noise in business data that it's like reading tea leaves, except no one can prove it because by the time they gather enough data they realize that the management and/or market dynamics have introduced even more noise so everyone keeps chugging along, blissfully unaware that they're indulging in statistical fantasies and reinforcing whatever beliefs they already hold.
hinkley 5y ago
Accidental or otherwise, we lost something. We had a situation where we had a three legged stool of product/marketing, QA, and Development. If any of the three were too far out of line, the other two could drag them back. But we killed QA, and Ken Schwaber gave us a treadmill to run on forever where structurally, nobody has time to think past the end of next month. So Product has been running progressively more roughshod over everything for the last 10 years. And since 50% of devs have 5 years or less of experience, that means most people don't know that things used to be better.
thewarrior 5y ago
I recently had a revelation that merely using a product can often reveal the organizational structure and the cultural incentives that went into making it. A company that is famous for being metrics driven and having teams that optimise for their own metrics is Facebook. You can tell that the product has "seams". Each page and widget feels like it was individually optimized and then stitched together. Overall it somehow feels off. Apple puts design on top and has a kind of dictatorial veto over all aspects of the product. Hence products made by Apple feel seamless and cohesive. They are also more polished because when you solely rely on metrics you make it as good as it needs to be and then stop.
- karaterobot 5y ago
  Sounds like you've independently discovered Conway's Law, one of the most terrible, true things you can say about software organizations: https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law
- andrewjf 5y ago
  Apple relies on intuition and experience and uses data to support decisions, or not (and data be regularly overruled). Rarely does pure data drive decisions. People coming from other companies usually fail if they can’t adapt to a “data second” mentality. I feel like Apple’s method is much better than blindly trying to interpret data and let that drive you over the cliff. Data isn’t a replacement for understanding of your product and users.
  - thewarrior 5y ago
    That's interesting. I was actually just guessing that Apple is not data driven. I would be grateful if you can share Apples general approach to product development.
    - andrewjf 5y ago
      The closest publicly available info is an HBR article about Apple’s organizational structure: https://hbr.org/2020/11/how-apple-is-organized-for-innovation https://hbr.org/2020/11/how-apple-is-organized-for-innovatio...

akiselev 5y ago

> After that experience, my view of metrics-driven product development is that it's a way to offload the burden of thinking too hard or understanding where you're going. When people talk about metrics-driven product development, I think of drivers who blindly follow their GPS' directions into a lake. There's also the flip side - metrics-driven development used to keep other people from thinking too hard and scrutinizing work. At my last company, many product decisions were justified using a litany of metrics including AB tests, which people just swallowed because "the data speaks for itself!" I once took a look at the app we were using to run the AB tests and not a single one had reached statistical significance. The company was basing its decisions on data considered garbage even by the AB app's naive p-value algorithm. I thought that after dozens of A/B tests at least one would be statistically significant just by dumb luck but no, not a single test had even the veneer of credibility.

kqr 5y ago
But what you're describing is noise-driven development. It's not an actual metric value until you've somewhat ruled out noise as the explanation.
- akiselev 5y ago
  I believe that's a distinction without a difference. Most scientists have a hard enough time understanding statistical significance in well controlled experiments and they are (ostensibly) trained for it. There's so much noise in business data that it's like reading tea leaves, except no one can prove it because by the time they gather enough data they realize that the management and/or market dynamics have introduced even more noise so everyone keeps chugging along, blissfully unaware that they're indulging in statistical fantasies and reinforcing whatever beliefs they already hold.

hinkley 5y ago

Accidental or otherwise, we lost something. We had a situation where we had a three legged stool of product/marketing, QA, and Development. If any of the three were too far out of line, the other two could drag them back. But we killed QA, and Ken Schwaber gave us a treadmill to run on forever where structurally, nobody has time to think past the end of next month. So Product has been running progressively more roughshod over everything for the last 10 years. And since 50% of devs have 5 years or less of experience, that means most people don't know that things used to be better.

thewarrior 5y ago

I recently had a revelation that merely using a product can often reveal the organizational structure and the cultural incentives that went into making it. A company that is famous for being metrics driven and having teams that optimise for their own metrics is Facebook. You can tell that the product has "seams". Each page and widget feels like it was individually optimized and then stitched together. Overall it somehow feels off. Apple puts design on top and has a kind of dictatorial veto over all aspects of the product. Hence products made by Apple feel seamless and cohesive. They are also more polished because when you solely rely on metrics you make it as good as it needs to be and then stop.

karaterobot 5y ago
Sounds like you've independently discovered Conway's Law, one of the most terrible, true things you can say about software organizations: https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law
andrewjf 5y ago
Apple relies on intuition and experience and uses data to support decisions, or not (and data be regularly overruled). Rarely does pure data drive decisions. People coming from other companies usually fail if they can’t adapt to a “data second” mentality. I feel like Apple’s method is much better than blindly trying to interpret data and let that drive you over the cliff. Data isn’t a replacement for understanding of your product and users.
- thewarrior 5y ago
  That's interesting. I was actually just guessing that Apple is not data driven. I would be grateful if you can share Apples general approach to product development.
  - andrewjf 5y ago
    The closest publicly available info is an HBR article about Apple’s organizational structure: https://hbr.org/2020/11/how-apple-is-organized-for-innovation https://hbr.org/2020/11/how-apple-is-organized-for-innovatio...