4 ms·

You're essentially arguing for qualitative data instead of quantitative, but both together is usually where the money is. I agree that qualitative analytics are

by pcstl 6y ago

You're essentially arguing for qualitative data instead of quantitative, but both together is usually where the money is. I agree that qualitative analytics are underestimatd because they're hard to do, but I also think that having quantitative analytics together with qualitative allows you to contextualize your numbers in ways that lead to insights you wouldn't have otherwise. Also, after you've already reached product-market fit, it's important to take your product to its "local maximum".

boplicity 6y ago
I think you may be right; after all, thinking critically about my story above, I did spend quite a lot of time learning about analytics and quantitative numbers. It could be this gave me an intuitive sense of what works, which I could then apply to the more creative thinking. I don't know. Either way, I'm grateful to make a living the way I do.
edmundsauto 6y ago
Qualitative data has another challenge - representativeness. It's very easy to do 10 user interviews and feel comfortable that you understand the market. Our brains lie to us all the time. Quantitative data lets you drill down into different dimensions. Because it is much easier to collect at scale (it's the sum of your users' interaction w/ your product, after all!), it's much easier to make representative decisions.
- hizxy 6y ago
  No it isn’t because the data will never tell you “why” people are doing something or not doing something. You can guess but you’ll never know why until you a) talk to users and b) watch them use your product. Qualitative research isn’t about statistical significance, it’s about deep insights. 10 user interviews will undercover 100 insights.
  - edmundsauto 6y ago
    What makes you think the 100 insights are actually insights, and not simply deriving from confirmation bias? To be clear, I support qual + quant user research (+ split testing). They are all tools that help influence product direction. I have, however, seen a lot of cases where qualitative research uncovers an insight that doesn't actually match real world behavior when we bake it into the product. Compare that to a split test, where I don't know why something is happening, but I am able to optimize against goals. It gives less insight but is more foolproof-actionable. With insight, you have to go insight -> improved_action. With testing, you just have the improved action. To me, qualitative research is crucial to build a model of the environment. These can be used to generate hypotheses, which you then split test. Any time I see someone take direct insight and build a product feature from it, I have a lot of questions about what alternate ideas they tested. And how to know the one that was developed is as close to optimal as you can get, given the fixed resources available.

by pcstl 6y ago

boplicity 6y ago

I think you may be right; after all, thinking critically about my story above, I did spend quite a lot of time learning about analytics and quantitative numbers. It could be this gave me an intuitive sense of what works, which I could then apply to the more creative thinking. I don't know. Either way, I'm grateful to make a living the way I do.

edmundsauto 6y ago

Qualitative data has another challenge - representativeness. It's very easy to do 10 user interviews and feel comfortable that you understand the market. Our brains lie to us all the time. Quantitative data lets you drill down into different dimensions. Because it is much easier to collect at scale (it's the sum of your users' interaction w/ your product, after all!), it's much easier to make representative decisions.

hizxy 6y ago
No it isn’t because the data will never tell you “why” people are doing something or not doing something. You can guess but you’ll never know why until you a) talk to users and b) watch them use your product. Qualitative research isn’t about statistical significance, it’s about deep insights. 10 user interviews will undercover 100 insights.
- edmundsauto 6y ago
  What makes you think the 100 insights are actually insights, and not simply deriving from confirmation bias? To be clear, I support qual + quant user research (+ split testing). They are all tools that help influence product direction. I have, however, seen a lot of cases where qualitative research uncovers an insight that doesn't actually match real world behavior when we bake it into the product. Compare that to a split test, where I don't know why something is happening, but I am able to optimize against goals. It gives less insight but is more foolproof-actionable. With insight, you have to go insight -> improved_action. With testing, you just have the improved action. To me, qualitative research is crucial to build a model of the environment. These can be used to generate hypotheses, which you then split test. Any time I see someone take direct insight and build a product feature from it, I have a lot of questions about what alternate ideas they tested. And how to know the one that was developed is as close to optimal as you can get, given the fixed resources available.