4 ms·

How to determine if your A/B test is statistically significant

87 points by zackattack 17y ago

Torn 17y ago
So, he's just explained Standard Deviation?
- showerst 17y ago
  You'd be amazed how many smart people don't have a decent grounding in statistics, so any easy reference to explain this is nice.
  - Torn 17y ago
    Fair point, and yes it's quite informative from that point of view. I didn't like his approach however, he seemed to be beating around the bush a lot with phrases like I'm here to rescue you with a statistically sound yet incredibly simple formula than giving a more honest 'hey, what you need is Standard Deviation, and here's how it works!'
    - smartbear 17y ago
      No Torn, you're still not using the correct statistic. THIS HAS NOTHING TO DO WITH STANDARD DEVIATION. Also, how can you say that the article isn't "honest," especially given the complete, precise explanation for the stats at the end.
      - Torn 17y ago
        Seems I'm haemorrhaging karma over this. I misjudged the method used (Pearson's Chi Square, as pointed out by hn user pibefision). Fine, that'll teach me to skim read and comment. My point above was that the author could have come out and labelled it as such in the article.
- pibefision 17y ago
  No, he explained Pearson's Chi Square.
araneae 17y ago
And besides that, she's clearly leading the witness (Hammy). See the outtakes: http://www.youtube.com/watch?v=VmZ7p3ExIc8 http://www.youtube.com/watch?v=VmZ7p3ExIc8
bmalicoat 17y ago
I like this rule of thumb a lot. It's very straight-forward and easy to remember so you can apply it quickly to see significance and then use a more rigorous method if necessary.
jsm386 17y ago
There are tons of tools that can test statistical significance for you: For example, http://www.splittestcalculator.com/ http://www.splittestcalculator.com/ or http://www.usereffect.com/split-test-calculator http://www.usereffect.com/split-test-calculator
steveplace 17y ago
But what about the vast majority of people who don't click either ad? That's the "ad impressions" that didn't lead to a click. Shouldn't those count somehow in the statistics? No, they shouldn't; those are "mistrials." I don't think those are mistrials. Clicks vs impressions is a separate test and should be treated as such.
- dkokelley 17y ago
  The test only counts if an even amount of impressions is given. The viewers aren't given a choice between ad A and B, they have a choice between A (or B) and nothing. I don't think the author explained this well enough. Obviously (to everyone here) the test is inconclusive if one ad is shown much more than the other.
wmeredith 17y ago
We can do this for you at the click of a button at 360voltage: http://360voltage.com/blog/2009/09/ab-split-testing-out-of-dark-ages/ http://360voltage.com/blog/2009/09/ab-split-testing-out-of-d... (For Free) </self promotion>
hadley 17y ago
The statistics are a little bit off - the Chi-squared isn't appropriate for the example he gives (sample size too small), although it would be fine in practice. What you really want in that situation is the exact binomial test.
- smartbear 17y ago
  Good point! Although with around N > 25 you're OK. See: http://faculty.chass.ncsu.edu/garson/PA765/binomial.htm http://faculty.chass.ncsu.edu/garson/PA765/binomial.htm

87 points by zackattack 17y ago

Torn 17y ago

So, he's just explained Standard Deviation?

showerst 17y ago
You'd be amazed how many smart people don't have a decent grounding in statistics, so any easy reference to explain this is nice.
- Torn 17y ago
  Fair point, and yes it's quite informative from that point of view. I didn't like his approach however, he seemed to be beating around the bush a lot with phrases like I'm here to rescue you with a statistically sound yet incredibly simple formula than giving a more honest 'hey, what you need is Standard Deviation, and here's how it works!'
  - smartbear 17y ago
    No Torn, you're still not using the correct statistic. THIS HAS NOTHING TO DO WITH STANDARD DEVIATION. Also, how can you say that the article isn't "honest," especially given the complete, precise explanation for the stats at the end.
    - Torn 17y ago
      Seems I'm haemorrhaging karma over this. I misjudged the method used (Pearson's Chi Square, as pointed out by hn user pibefision). Fine, that'll teach me to skim read and comment. My point above was that the author could have come out and labelled it as such in the article.
pibefision 17y ago
No, he explained Pearson's Chi Square.

araneae 17y ago

And besides that, she's clearly leading the witness (Hammy). See the outtakes: http://www.youtube.com/watch?v=VmZ7p3ExIc8 http://www.youtube.com/watch?v=VmZ7p3ExIc8

bmalicoat 17y ago

I like this rule of thumb a lot. It's very straight-forward and easy to remember so you can apply it quickly to see significance and then use a more rigorous method if necessary.

jsm386 17y ago

There are tons of tools that can test statistical significance for you: For example, http://www.splittestcalculator.com/ http://www.splittestcalculator.com/ or http://www.usereffect.com/split-test-calculator http://www.usereffect.com/split-test-calculator

steveplace 17y ago

But what about the vast majority of people who don't click either ad? That's the "ad impressions" that didn't lead to a click. Shouldn't those count somehow in the statistics? No, they shouldn't; those are "mistrials." I don't think those are mistrials. Clicks vs impressions is a separate test and should be treated as such.

dkokelley 17y ago
The test only counts if an even amount of impressions is given. The viewers aren't given a choice between ad A and B, they have a choice between A (or B) and nothing. I don't think the author explained this well enough. Obviously (to everyone here) the test is inconclusive if one ad is shown much more than the other.

wmeredith 17y ago

We can do this for you at the click of a button at 360voltage: http://360voltage.com/blog/2009/09/ab-split-testing-out-of-dark-ages/ http://360voltage.com/blog/2009/09/ab-split-testing-out-of-d... (For Free) </self promotion>

hadley 17y ago

The statistics are a little bit off - the Chi-squared isn't appropriate for the example he gives (sample size too small), although it would be fine in practice. What you really want in that situation is the exact binomial test.

smartbear 17y ago
Good point! Although with around N > 25 you're OK. See: http://faculty.chass.ncsu.edu/garson/PA765/binomial.htm http://faculty.chass.ncsu.edu/garson/PA765/binomial.htm