4 ms·

One problem: he never discuses how to analyze the results. That is a pretty big missing piece for an "ultimate" guide. There should be some discussion of statis

by whyenot 16y ago

One problem: he never discuses how to analyze the results. That is a pretty big missing piece for an "ultimate" guide. There should be some discussion of statistical tests. Yes, it may seem obvious, but there have been A/B case studies posted to HN where nobody performed any inferential statistics at all. If you don't do any statistics, you don't have all the information you need to make a decision. Results that may look conclusive may not be conclusive at all.

paulgb 16y ago
By inferential statistics, are you referring to confidence intervals and tests for significance, or is there more to it than that?
- nostrademons 16y ago
  There's a huge black art to interpreting data. It's not just confidence intervals and significance tests: you also need to watch very closely for any sources of bias in your data. Different user populations, unexpected feature interactions, bugs in your logging code, changes in the site midway through your experiment period, etc.
- whyenot 16y ago
  Drawing conclusions from the dataset as opposed to descriptive statistics such as the mean, variance, etc.
Vindexus 16y ago
I'm pretty lost when it comes to statistics. Are you talking about not having enough data to make a conclusion, or something else?
- whyenot 16y ago
  Suppose in A/B testing A has a higher "conversion rate" than B. Based on the data you collected, you conclude that A is the better option. Sounds good, BUT if you did not take the time to actually do some statistical hypothesis testing, you could be making a mistake. What are the chances that A and B will have exactly the same conversion rate? For any reasonably large set of data, it is very close to 0. That means that A > B or B > A no matter what A and B actually are. The difference you observed between A and B could be real or it could be noise. Various statistical tests[1] can help you in deciding between the two possibilities. Flip a coin twice. Lets say you get two heads. If you then conclude that for this coin heads are much more likely than tails, that would be wrong. This is the sort of mistake statistical tests can help you avoid. [1] I'm being intentionally vague here because what statistical tests you should use depends on what A and B are, what assumptions you are willing to make, sample size, and other factors. A good starting place is probably to use a chi-square test.
  - sesqu 16y ago
    Switching when there is no effect is harmless, from the perspective of the product. It only hurts your takeaways, and if you're doing A/B, you're conceding that your previous takeaways weren't the final word anyway. Now, if switching is expensive for some reason, and your A/B isn't as conclusive as you'd need, there's a pretty good chance your change resistance will catch that. So even then it's probably not a big deal.

by whyenot 16y ago

paulgb 16y ago

By inferential statistics, are you referring to confidence intervals and tests for significance, or is there more to it than that?

nostrademons 16y ago
There's a huge black art to interpreting data. It's not just confidence intervals and significance tests: you also need to watch very closely for any sources of bias in your data. Different user populations, unexpected feature interactions, bugs in your logging code, changes in the site midway through your experiment period, etc.
whyenot 16y ago
Drawing conclusions from the dataset as opposed to descriptive statistics such as the mean, variance, etc.

Vindexus 16y ago

I'm pretty lost when it comes to statistics. Are you talking about not having enough data to make a conclusion, or something else?

whyenot 16y ago
Suppose in A/B testing A has a higher "conversion rate" than B. Based on the data you collected, you conclude that A is the better option. Sounds good, BUT if you did not take the time to actually do some statistical hypothesis testing, you could be making a mistake. What are the chances that A and B will have exactly the same conversion rate? For any reasonably large set of data, it is very close to 0. That means that A > B or B > A no matter what A and B actually are. The difference you observed between A and B could be real or it could be noise. Various statistical tests[1] can help you in deciding between the two possibilities. Flip a coin twice. Lets say you get two heads. If you then conclude that for this coin heads are much more likely than tails, that would be wrong. This is the sort of mistake statistical tests can help you avoid. [1] I'm being intentionally vague here because what statistical tests you should use depends on what A and B are, what assumptions you are willing to make, sample size, and other factors. A good starting place is probably to use a chi-square test.
- sesqu 16y ago
  Switching when there is no effect is harmless, from the perspective of the product. It only hurts your takeaways, and if you're doing A/B, you're conceding that your previous takeaways weren't the final word anyway. Now, if switching is expensive for some reason, and your A/B isn't as conclusive as you'd need, there's a pretty good chance your change resistance will catch that. So even then it's probably not a big deal.