5 ms·
Great article. This hits close to home as a product developer. I'm certain this phenomena happens in "data driven" product decisions. Top of the funnel, we have
by code_biologist 7y ago
Great article. This hits close to home as a product developer. I'm certain this phenomena happens in "data driven" product decisions. Top of the funnel, we have the numbers to get statistically sound outcomes on A/B tests quickly. I work on a B2B product (high value, low volume sales cycle) and bottom of the funnel or internal sales process experimentation we try our best at is sketchy and hard to do. Experimental design is hard trying to get adequate power.
Any HNers have tips for working in low-data regimes (N=low hundreds)? Is there some magic Bayesian angle I don't know about?
We haven't systemically looked at effect sizes and this article is a good push for me to dig into that.
- jack_pp 7y agoMore data = better prediction. If you have a million users you can get few data per user but lots of data in total. If you have 100 users you can get a lot of data on each one by direct contact
- AstralStorm 7y agoThat always depends on the goal. Locking yourself into 100 early adopters may or may not be prudent. Main thing to answer is how representative your sample of target users is. The deeper the questions asked, less likely it is.
- eru 7y agoThere's money on the line for companies. So directly or indirectly developer are going to be paranoid about fooling themselves. Like you demonstrate here! Alas, some of the social sciences didn't use to be so honest. See this infamous piece about 'methodological terrorism': http://datacolada.org/wp-content/uploads/2016/09/Fiske-presidential-guest-column_APS-Observer_copy-edited.pdf http://datacolada.org/wp-content/uploads/2016/09/Fiske-presi... (or https://www.thecut.com/2016/10/inside-psychologys-methodological-terrorism-debate.html https://www.thecut.com/2016/10/inside-psychologys-methodolog... for an overview).