3 ms·

I used http://www.abtestcalculator.com/ http://www.abtestcalculator.com/ and entered 3000 participants -> 28 conversions and 3000 participants -> 39 conversions

by bemmu 8y ago

I used http://www.abtestcalculator.com/ http://www.abtestcalculator.com/ and entered 3000 participants -> 28 conversions and 3000 participants -> 39 conversions. I neglected to record how many views each version had, but should be at least 3000 since the conversion ratio is about 0.5 - 1%.

no_gravity 8y ago
The statement I see on that page is "There is a 91% chance that Variation A has a higher conversion rate". I am not sure how to interpret that. We would have to dive into the GitHub repo and figure out which test it performs I guess.
- bemmu 8y ago
  Looks like I misunderstood the result, I will change the post to reflect this.
- eutectic 8y ago
  You get 91% if you put a uniform prior on the proportion coming from each alternative.
- popinman322 8y ago
  It looks like the difference of two beta distributions based on the visualization. So, assuming a uniform prior and updating with 39/3000 and 28/3000 conversions the difference between the two distributions is greater than zero 91% of the time. It's only guaranteed to be above zero at about the 80% credible interval, and since we started with an uninformed prior that'd be about p=.2? I'm open to correction here.
artiscode 8y ago
Resource provided uses a very naive approach to determining the outcome of an AB test. It's not accurate, given the very small numbers.
- bemmu 8y ago
  Yes, I don't have much data to work with, and was also surprised that the calculator considered this significant. But even without significance, I assume it still makes sense to go with the winner?
  - no_gravity 8y ago
    Does the calculator really use the word "significant"? I don't see it. I am not sure how to interpret the language it uses. As for going with the winner: Yes, if the test result (39/28) is the only information you have and there are only two choices (go with winner / go with loser) then it makes sense to go with the winner.
funkaster 8y ago
If you had 3000 visits, shouldn't that be 1500 -> 28 and 1500 -> 39? (assuming you're doing a uniform split of both groups)
rchaud 8y ago
If you have Google Analyics on your site, you can use the Unique Pageviews metric for each of the two page variations and use that as the metric, instead of arbitrarily assigning 3000 views to both.

by bemmu 8y ago

no_gravity 8y ago

The statement I see on that page is "There is a 91% chance that Variation A has a higher conversion rate". I am not sure how to interpret that. We would have to dive into the GitHub repo and figure out which test it performs I guess.

bemmu 8y ago
Looks like I misunderstood the result, I will change the post to reflect this.
eutectic 8y ago
You get 91% if you put a uniform prior on the proportion coming from each alternative.
popinman322 8y ago
It looks like the difference of two beta distributions based on the visualization. So, assuming a uniform prior and updating with 39/3000 and 28/3000 conversions the difference between the two distributions is greater than zero 91% of the time. It's only guaranteed to be above zero at about the 80% credible interval, and since we started with an uninformed prior that'd be about p=.2? I'm open to correction here.

artiscode 8y ago

Resource provided uses a very naive approach to determining the outcome of an AB test. It's not accurate, given the very small numbers.

bemmu 8y ago
Yes, I don't have much data to work with, and was also surprised that the calculator considered this significant. But even without significance, I assume it still makes sense to go with the winner?
- no_gravity 8y ago
  Does the calculator really use the word "significant"? I don't see it. I am not sure how to interpret the language it uses. As for going with the winner: Yes, if the test result (39/28) is the only information you have and there are only two choices (go with winner / go with loser) then it makes sense to go with the winner.

funkaster 8y ago

If you had 3000 visits, shouldn't that be 1500 -> 28 and 1500 -> 39? (assuming you're doing a uniform split of both groups)

rchaud 8y ago

If you have Google Analyics on your site, you can use the Unique Pageviews metric for each of the two page variations and use that as the metric, instead of arbitrarily assigning 3000 views to both.