4 ms·

Genuine question – what's the alternative to trying? Being data-driven is definitely hard but imho teams should be commended for striving towards it. Even impe

Genuine question – what's the alternative to trying? Being data-driven is definitely hard but imho teams should be commended for striving towards it. Even imperfect shifts towards data-driven decision making can be valuable as well (eg maybe you have great data for optimizing your onboarding flow, even if knowledge of which features most improve retention or database performance are still black boxes).

pmichaud 6y ago
I think part of the issue is that rampant innumeracy makes half-assed attempts at being data-driven actually worse than nothing. The reason is that without clean data we fall back to the old standard: gut feel + pet theories. Which of course is often not great, but then again almost always has more than zero correlation with reality than random. Poorly executed data collection and analysis on the other hand, I think drives many teams at full speed right off a cliff. All the randomness of flying blind with the confidence of feeling certain.
- Toenex 6y ago
  Being data driven can be as simple as sales telling you how many people are asking for a feature as opposed to "everybody".
  - hansvm 6y ago
    Even in that simple of an example, we lack the data we need for that number to matter much, and the fact that it's a concrete statistic can easily lead to prioritizing one feature or another using that imperfect data as a cudgel. In particular, you'd probably want to know how a feature would impact future sales, survivorship curves for current customers, and survivorship curves for whichever kind of customer would sign on with the new feature (treating a single feature in isolation for simplicity). This is especially important when comparing multiple features to put on the roadmap because it's easy for one idea to be simple to imagine and better than the status quo (hence asked for by many customers) but be nowhere near an optimal solution to whichever problem is being solved. Having that kind of customer insight is better than not having it, as long as such data is used appropriately -- without additional data it can't do much more that guide or refine gut feelings and insights, and attempting to do otherwise is a recipe for an inferior product.
    - ochronus 6y ago
      Great point, and the post never intended to talk about this topic. I think there's great merit to 'gut feeling' from field experts, even if they cannot support it with data at the moment, maybe because the nature of the thing is exploratory at that certain stage or it's simply too chaotic. Experience and intuition coming from experts tenured in relevant areas is priceless - but it doesn't have to come with hierarchy or power based authority. When people naturally respect someone they will listen. No need to force it.
ochronus 6y ago
I clearly failed with the post (by not elaborating) on this point - I never meant forcing fake data where it's not possible. I'd still rather have this conversation: A: We should definitely do X! B: Why? Why not Y? A: Because I'm the authority in the room B: Well we talked to sales and customers and Y clearly has Z impact, on the other hand we saw no evidence of customers needing X. than A: We should definitely do X! B: Why? Why not Y? A: Because I'm the authority in the room B: Whelp, ok I guess

pmichaud 6y ago

I think part of the issue is that rampant innumeracy makes half-assed attempts at being data-driven actually worse than nothing. The reason is that without clean data we fall back to the old standard: gut feel + pet theories. Which of course is often not great, but then again almost always has more than zero correlation with reality than random. Poorly executed data collection and analysis on the other hand, I think drives many teams at full speed right off a cliff. All the randomness of flying blind with the confidence of feeling certain.

Toenex 6y ago
Being data driven can be as simple as sales telling you how many people are asking for a feature as opposed to "everybody".
- hansvm 6y ago
  Even in that simple of an example, we lack the data we need for that number to matter much, and the fact that it's a concrete statistic can easily lead to prioritizing one feature or another using that imperfect data as a cudgel. In particular, you'd probably want to know how a feature would impact future sales, survivorship curves for current customers, and survivorship curves for whichever kind of customer would sign on with the new feature (treating a single feature in isolation for simplicity). This is especially important when comparing multiple features to put on the roadmap because it's easy for one idea to be simple to imagine and better than the status quo (hence asked for by many customers) but be nowhere near an optimal solution to whichever problem is being solved. Having that kind of customer insight is better than not having it, as long as such data is used appropriately -- without additional data it can't do much more that guide or refine gut feelings and insights, and attempting to do otherwise is a recipe for an inferior product.
  - ochronus 6y ago
    Great point, and the post never intended to talk about this topic. I think there's great merit to 'gut feeling' from field experts, even if they cannot support it with data at the moment, maybe because the nature of the thing is exploratory at that certain stage or it's simply too chaotic. Experience and intuition coming from experts tenured in relevant areas is priceless - but it doesn't have to come with hierarchy or power based authority. When people naturally respect someone they will listen. No need to force it.

ochronus 6y ago

I clearly failed with the post (by not elaborating) on this point - I never meant forcing fake data where it's not possible. I'd still rather have this conversation: A: We should definitely do X! B: Why? Why not Y? A: Because I'm the authority in the room B: Well we talked to sales and customers and Y clearly has Z impact, on the other hand we saw no evidence of customers needing X. than A: We should definitely do X! B: Why? Why not Y? A: Because I'm the authority in the room B: Whelp, ok I guess