3 ms·
I know more than a bit about statistics and A/B testing. My first reaction is that asking for more information from a limited sample set does not actually give
by btilly 14y ago
I know more than a bit about statistics and A/B testing. My first reaction is that asking for more information from a limited sample set does not actually give you the power to do more powerful statistics. Claiming otherwise is a bad sign.
Secondly a lot of what A/B testing gets at is subconscious reaction, not conscious reactions. Consciously when we're asked a question and are paying attention, we are poor judges of how we're going to react to, say, a larger button to click on. But when we're actually using the site, we notice and hit the larger button.
And thirdly, if you're actually going to seek out a small sample of people for detailed information capture, do it in person. I can't stress this enough. If you think up questions in a vacuum it is hard to ask the right questions. That is part of what makes A/B testing so frustrating. But if you have a conversation and are observant, you'll quickly generate good ideas.
- jiaaro 14y agoI agree that sample size is important and that's why you still need 200+ responses. It's a hybrid between a survey and an A/B test. You ask a question, but you get to find out about more than just the one variation that the person is served up. I think it's fair to say that there is more info to be gathered from each individual than time-on-page, scroll behavior, and converted-or-didnt If you like, you can think of this as a way to get more resolution from a survey-typed question