5 ms·

"Push for a data-driven culture." Yep, stopped giving any credence to this article after this line. In practice, data-driven is extremely hard. Most A/B testi

"Push for a data-driven culture." Yep, stopped giving any credence to this article after this line. In practice, data-driven is extremely hard. Most A/B testing is blindness dressed in nice clothes. Data-driven is only cost-effective at some (large) scale, because you'll need actual experts and infrastructure.

jaclaz 6y ago
>Most A/B testing is blindness dressed in nice clothes. ... and harassing you. Excellent definition.
- ochronus 6y ago
  A/B testing is but one way of gathering data. Also, it usually comes after the decision to actually invest in building _something_ out (even if it's only a prototype/MVP). There are other types of data such as sizing the opportunity based on market research, competitor analysis or simply asking your customers - among many other. This shouldn't be trusted blindly either, sure. Never said that.
nautilus12 6y ago
Agree, it's just mumbo jumbo to most people. Being really data driven in many realms falls into the category of Complicated or Chaos (in terms of the Cynefin framework) so often times being truly scientifically data driven is a matter of a PhD thesis. The only true recourse many companies have is intuition and trial and error which according to Cynefin is the most effective means of operating in the chaos realm
cpufry 6y ago
my xp at big software corps has been that technical decisions are often heavily data driven. product decisions are rarely so. i think about how there will deep dive root cause analysis for technical misses, but there is nothing like that for product misses.
- alonmower 6y ago
  There should be analyses of product misses in a well functioning product org, though getting to a concrete “root cause” can be a lot harder and may not need to be the goal. You should at least be trying to understand, to the extent possible, why something didn’t work and then using that to better understand the problem you’re were originally trying to solve.
staysaasy 6y ago
Genuine question – what's the alternative to trying? Being data-driven is definitely hard but imho teams should be commended for striving towards it. Even imperfect shifts towards data-driven decision making can be valuable as well (eg maybe you have great data for optimizing your onboarding flow, even if knowledge of which features most improve retention or database performance are still black boxes).
- pmichaud 6y ago
  I think part of the issue is that rampant innumeracy makes half-assed attempts at being data-driven actually worse than nothing. The reason is that without clean data we fall back to the old standard: gut feel + pet theories. Which of course is often not great, but then again almost always has more than zero correlation with reality than random. Poorly executed data collection and analysis on the other hand, I think drives many teams at full speed right off a cliff. All the randomness of flying blind with the confidence of feeling certain.
  - Toenex 6y ago
    Being data driven can be as simple as sales telling you how many people are asking for a feature as opposed to "everybody".
    - hansvm 6y ago
      Even in that simple of an example, we lack the data we need for that number to matter much, and the fact that it's a concrete statistic can easily lead to prioritizing one feature or another using that imperfect data as a cudgel. In particular, you'd probably want to know how a feature would impact future sales, survivorship curves for current customers, and survivorship curves for whichever kind of customer would sign on with the new feature (treating a single feature in isolation for simplicity). This is especially important when comparing multiple features to put on the roadmap because it's easy for one idea to be simple to imagine and better than the status quo (hence asked for by many customers) but be nowhere near an optimal solution to whichever problem is being solved. Having that kind of customer insight is better than not having it, as long as such data is used appropriately -- without additional data it can't do much more that guide or refine gut feelings and insights, and attempting to do otherwise is a recipe for an inferior product.
choeger 6y ago
Data-driven works well for cheap services with many customers. Try it with expensive services with few customers and your experiment quickly becomes statistically worthless and expensive. Also, data-driven does never work for a product. If you sell a product, your customers do not want to be monitored. They also do not want to randomly get an inferior version of the very product they paid for just to satisfy your research goal. So your only realistic test case would be to offer a second product and see how it sells. Or to pay for expensive surveys.
baby 6y ago
What bothers me with this kind Of culture is that it quickly becomes a gatekeeper for ideas you don’t like. Like an idea? Push for it without data. Don’t like an idea? Ask for more data.
- jayd16 6y ago
  This is what the author is saying whether they realize it or not. This is the advice for dealing with a ZEBRA. Data as a cudgel.
- ochronus 6y ago
  Excellent point and potentially good material for a separate post - I totally agree. On that note almost any kind of culture can be (mis)used to create gatekeepers.

jaclaz 6y ago

>Most A/B testing is blindness dressed in nice clothes. ... and harassing you. Excellent definition.

ochronus 6y ago
A/B testing is but one way of gathering data. Also, it usually comes after the decision to actually invest in building _something_ out (even if it's only a prototype/MVP). There are other types of data such as sizing the opportunity based on market research, competitor analysis or simply asking your customers - among many other. This shouldn't be trusted blindly either, sure. Never said that.

nautilus12 6y ago

Agree, it's just mumbo jumbo to most people. Being really data driven in many realms falls into the category of Complicated or Chaos (in terms of the Cynefin framework) so often times being truly scientifically data driven is a matter of a PhD thesis. The only true recourse many companies have is intuition and trial and error which according to Cynefin is the most effective means of operating in the chaos realm

cpufry 6y ago

my xp at big software corps has been that technical decisions are often heavily data driven. product decisions are rarely so. i think about how there will deep dive root cause analysis for technical misses, but there is nothing like that for product misses.

alonmower 6y ago
There should be analyses of product misses in a well functioning product org, though getting to a concrete “root cause” can be a lot harder and may not need to be the goal. You should at least be trying to understand, to the extent possible, why something didn’t work and then using that to better understand the problem you’re were originally trying to solve.

staysaasy 6y ago

Genuine question – what's the alternative to trying? Being data-driven is definitely hard but imho teams should be commended for striving towards it. Even imperfect shifts towards data-driven decision making can be valuable as well (eg maybe you have great data for optimizing your onboarding flow, even if knowledge of which features most improve retention or database performance are still black boxes).

pmichaud 6y ago
I think part of the issue is that rampant innumeracy makes half-assed attempts at being data-driven actually worse than nothing. The reason is that without clean data we fall back to the old standard: gut feel + pet theories. Which of course is often not great, but then again almost always has more than zero correlation with reality than random. Poorly executed data collection and analysis on the other hand, I think drives many teams at full speed right off a cliff. All the randomness of flying blind with the confidence of feeling certain.
- Toenex 6y ago
  Being data driven can be as simple as sales telling you how many people are asking for a feature as opposed to "everybody".
  - hansvm 6y ago
    Even in that simple of an example, we lack the data we need for that number to matter much, and the fact that it's a concrete statistic can easily lead to prioritizing one feature or another using that imperfect data as a cudgel. In particular, you'd probably want to know how a feature would impact future sales, survivorship curves for current customers, and survivorship curves for whichever kind of customer would sign on with the new feature (treating a single feature in isolation for simplicity). This is especially important when comparing multiple features to put on the roadmap because it's easy for one idea to be simple to imagine and better than the status quo (hence asked for by many customers) but be nowhere near an optimal solution to whichever problem is being solved. Having that kind of customer insight is better than not having it, as long as such data is used appropriately -- without additional data it can't do much more that guide or refine gut feelings and insights, and attempting to do otherwise is a recipe for an inferior product.

choeger 6y ago

Data-driven works well for cheap services with many customers. Try it with expensive services with few customers and your experiment quickly becomes statistically worthless and expensive. Also, data-driven does never work for a product. If you sell a product, your customers do not want to be monitored. They also do not want to randomly get an inferior version of the very product they paid for just to satisfy your research goal. So your only realistic test case would be to offer a second product and see how it sells. Or to pay for expensive surveys.

baby 6y ago

What bothers me with this kind Of culture is that it quickly becomes a gatekeeper for ideas you don’t like. Like an idea? Push for it without data. Don’t like an idea? Ask for more data.

jayd16 6y ago
This is what the author is saying whether they realize it or not. This is the advice for dealing with a ZEBRA. Data as a cudgel.
ochronus 6y ago
Excellent point and potentially good material for a separate post - I totally agree. On that note almost any kind of culture can be (mis)used to create gatekeepers.