3 ms·

During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks

During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. Is it? This seems a small population to settle on a clear winner. Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it). I think this means that in a world where it makes no difference which design is used, you would get a result as significant as this 22% of the time. An alternative is the Adjusted Wald method. You can try it online here: https://measuringu.com/wald/ https://measuringu.com/wald/ Which gives some confidence intervals which also range from "could be better" to "could be worse". Even when you reduce the confidence level from the typical 95% to 90%. a quick check with an A/B testing calculator even says that this result has significance (~90% likely) Which calculator was that?

ta2233221 8y ago
Reminds me of: "A/A Testing: How I increased conversions 300% by doing absolutely nothing" <https://kadavy.net/blog/posts/aa-testing/> https://kadavy.net/blog/posts/aa-testing/>
deleted 8y ago
[deleted]
olavolav 8y ago
Whatever test you do, you would need to know the total number of visitors in each group, right? And unless I missed it the article doesn't state those numbers. Intuitively, the numbers you quoted would be more significant the bigger the test and control groups are.
- kqr 8y ago
  That's not very intuitive for me. Let's do some limit analysis: imagine the groups were one million sessions each, but the convertions in the groups were only one and two people respectively. Wouldn't this result seem like the result of random chance? The conversion rate is basically one in a million in both cases.
bemmu 8y ago
I used http://www.abtestcalculator.com/ http://www.abtestcalculator.com/ and entered 3000 participants -> 28 conversions and 3000 participants -> 39 conversions. I neglected to record how many views each version had, but should be at least 3000 since the conversion ratio is about 0.5 - 1%.
- no_gravity 8y ago
  The statement I see on that page is "There is a 91% chance that Variation A has a higher conversion rate". I am not sure how to interpret that. We would have to dive into the GitHub repo and figure out which test it performs I guess.
  - bemmu 8y ago
    Looks like I misunderstood the result, I will change the post to reflect this.
  - eutectic 8y ago
    You get 91% if you put a uniform prior on the proportion coming from each alternative.
  - popinman322 8y ago
    It looks like the difference of two beta distributions based on the visualization. So, assuming a uniform prior and updating with 39/3000 and 28/3000 conversions the difference between the two distributions is greater than zero 91% of the time. It's only guaranteed to be above zero at about the 80% credible interval, and since we started with an uninformed prior that'd be about p=.2? I'm open to correction here.
- artiscode 8y ago
  Resource provided uses a very naive approach to determining the outcome of an AB test. It's not accurate, given the very small numbers.
  - bemmu 8y ago
muraiki 8y ago
That's not how you use prop.test. What you've tested using that invocation is the null hypothesis that the underlying probability of 39/67 is 0.5. If you want to perform a test of a difference of two proportions, you need to do: prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions)) I don't have experience with A/B testing, so I'm not sure if this is typically or best handled using this particular statistical test. Edit: The first parameter should be c(39, 28), meaning the total conversions in each group. I have no excuse beyond being tired. Edit 2: To clarify, I think he should still use the two-sample form of prop.test, especially since we did not know at the time of his posting that the sample sizes are equal.
- no_gravity 8y ago
  What you've tested using that invocation is the null hypothesis that the underlying probability of 39/67 is 0.5. Isn't that equivalent to my interpretation of the test result? "In a world where it makes no difference which design is used, you would get a result as significant as this 22% of the time". If you want to perform a test of a difference of two proportions, you need to do: prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions)) Do you mean c(39,28)? Because group_a had 39 hits and group_b hat 28. Doing so with the group sizes Bemmu stated (3000/3000) also gives me a p value of 0.22. As long as the group sizes are equal, the test is not very sensitive to the sizes.
  - muraiki 8y ago
    I think there is a difference in the approaches, given that the Chi-squared test statistic for the two-sample version is ~1.52, while for your one-sample version it is ~1.81. If group size doesn't matter and if you're justified in adding up successes as you have, I'd expect the test statistics to be nearly the same. Edit: I'd expect them to be nearly the same since the Chi-squared distributions would be parameterized similarly, so if we have similar results, we should see similar test statistics. Maybe my reasoning here is incorrect though!
rwilson4 8y ago
Using the techniques described on my blog [0], which are ideal for KPIs like conversion rates and small sample sizes (since no Gaussian approximation is made), I get a p-value of 0.177, which is not significant. The observed treatment effect is a 36.8% lift in conversion rate, but a confidence interval on this effect has endpoints -10.7% and +97.4%. Anything in that range would be considered consistent with the observed result at a 0.05 significance threshold. With 6000 total impressions and a 50/50 split, the experiment is only able reliably to detect a 74% lift in conversion rate (with power = 80%). If you want to rigorously determine the impact, decide what effect size you hope to see. Use a power calculator to decide the sample size needed to detect that effect size. Administer the test, waiting to acquire the planned sample size. When analyzing, be sure to compute a p-value and a confidence interval on the treatment effect. [0] https://www.adventuresinwhy.com/post/ab-testing-random-sampling/ https://www.adventuresinwhy.com/post/ab-testing-random-sampl...
codetrotter 8y ago
They said the following in the beginning though: > For example if you want to test a tweak that results in 5% more conversions, you need about 3000 sales to detect it! For Candy Japan this would mean waiting for about 10 years for the test to complete. But they want to do something still to try and improve sales. Seems reasonable even if not scientific.
- cle 8y ago
  This is not the right approach to take then. There are lots of other approaches to decision making outside of hypothesis testing, use them! This is not an appropriate use of hypothesis testing and can very much lead you toward making the WRONG decision. For example, with such small numbers, there isn't much value in aggregate statistics. It would take a day or two to go through each one individually and see what happened, and you'd probably learn way more about your customers.
closed 8y ago
I don't think that's the correct use of prop.test for this question. When you give it two numbers (success, total), it tests against the null that chance of success is 50%. Here, we want to test whether p(success|cond) differs across conditions, not whether prob(cond|success) is 50%. This distinction is important because when p(success|cond) for some cond is low, its variance is also very small, but prop.test(39,67) doesn't reflect this. That could be 67 successes out of a small sample (and high chance of success), or out of a huge sample (and low chance of success). Edit: whoops, I didn't notice other comments point out this issue

ta2233221 8y ago

Reminds me of: "A/A Testing: How I increased conversions 300% by doing absolutely nothing" <https://kadavy.net/blog/posts/aa-testing/> https://kadavy.net/blog/posts/aa-testing/>

deleted 8y ago

[deleted]

olavolav 8y ago

Whatever test you do, you would need to know the total number of visitors in each group, right? And unless I missed it the article doesn't state those numbers. Intuitively, the numbers you quoted would be more significant the bigger the test and control groups are.

kqr 8y ago
That's not very intuitive for me. Let's do some limit analysis: imagine the groups were one million sessions each, but the convertions in the groups were only one and two people respectively. Wouldn't this result seem like the result of random chance? The conversion rate is basically one in a million in both cases.

bemmu 8y ago

I used http://www.abtestcalculator.com/ http://www.abtestcalculator.com/ and entered 3000 participants -> 28 conversions and 3000 participants -> 39 conversions. I neglected to record how many views each version had, but should be at least 3000 since the conversion ratio is about 0.5 - 1%.

no_gravity 8y ago
The statement I see on that page is "There is a 91% chance that Variation A has a higher conversion rate". I am not sure how to interpret that. We would have to dive into the GitHub repo and figure out which test it performs I guess.
- bemmu 8y ago
  Looks like I misunderstood the result, I will change the post to reflect this.
- eutectic 8y ago
  You get 91% if you put a uniform prior on the proportion coming from each alternative.
- popinman322 8y ago
  It looks like the difference of two beta distributions based on the visualization. So, assuming a uniform prior and updating with 39/3000 and 28/3000 conversions the difference between the two distributions is greater than zero 91% of the time. It's only guaranteed to be above zero at about the 80% credible interval, and since we started with an uninformed prior that'd be about p=.2? I'm open to correction here.
artiscode 8y ago
Resource provided uses a very naive approach to determining the outcome of an AB test. It's not accurate, given the very small numbers.
- bemmu 8y ago

muraiki 8y ago

That's not how you use prop.test. What you've tested using that invocation is the null hypothesis that the underlying probability of 39/67 is 0.5. If you want to perform a test of a difference of two proportions, you need to do: prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions)) I don't have experience with A/B testing, so I'm not sure if this is typically or best handled using this particular statistical test. Edit: The first parameter should be c(39, 28), meaning the total conversions in each group. I have no excuse beyond being tired. Edit 2: To clarify, I think he should still use the two-sample form of prop.test, especially since we did not know at the time of his posting that the sample sizes are equal.

no_gravity 8y ago
What you've tested using that invocation is the null hypothesis that the underlying probability of 39/67 is 0.5. Isn't that equivalent to my interpretation of the test result? "In a world where it makes no difference which design is used, you would get a result as significant as this 22% of the time". If you want to perform a test of a difference of two proportions, you need to do: prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions)) Do you mean c(39,28)? Because group_a had 39 hits and group_b hat 28. Doing so with the group sizes Bemmu stated (3000/3000) also gives me a p value of 0.22. As long as the group sizes are equal, the test is not very sensitive to the sizes.
- muraiki 8y ago
  I think there is a difference in the approaches, given that the Chi-squared test statistic for the two-sample version is ~1.52, while for your one-sample version it is ~1.81. If group size doesn't matter and if you're justified in adding up successes as you have, I'd expect the test statistics to be nearly the same. Edit: I'd expect them to be nearly the same since the Chi-squared distributions would be parameterized similarly, so if we have similar results, we should see similar test statistics. Maybe my reasoning here is incorrect though!

rwilson4 8y ago

Using the techniques described on my blog [0], which are ideal for KPIs like conversion rates and small sample sizes (since no Gaussian approximation is made), I get a p-value of 0.177, which is not significant. The observed treatment effect is a 36.8% lift in conversion rate, but a confidence interval on this effect has endpoints -10.7% and +97.4%. Anything in that range would be considered consistent with the observed result at a 0.05 significance threshold. With 6000 total impressions and a 50/50 split, the experiment is only able reliably to detect a 74% lift in conversion rate (with power = 80%). If you want to rigorously determine the impact, decide what effect size you hope to see. Use a power calculator to decide the sample size needed to detect that effect size. Administer the test, waiting to acquire the planned sample size. When analyzing, be sure to compute a p-value and a confidence interval on the treatment effect. [0] https://www.adventuresinwhy.com/post/ab-testing-random-sampling/ https://www.adventuresinwhy.com/post/ab-testing-random-sampl...

codetrotter 8y ago

They said the following in the beginning though: > For example if you want to test a tweak that results in 5% more conversions, you need about 3000 sales to detect it! For Candy Japan this would mean waiting for about 10 years for the test to complete. But they want to do something still to try and improve sales. Seems reasonable even if not scientific.

cle 8y ago
This is not the right approach to take then. There are lots of other approaches to decision making outside of hypothesis testing, use them! This is not an appropriate use of hypothesis testing and can very much lead you toward making the WRONG decision. For example, with such small numbers, there isn't much value in aggregate statistics. It would take a day or two to go through each one individually and see what happened, and you'd probably learn way more about your customers.

closed 8y ago

I don't think that's the correct use of prop.test for this question. When you give it two numbers (success, total), it tests against the null that chance of success is 50%. Here, we want to test whether p(success|cond) differs across conditions, not whether prob(cond|success) is 50%. This distinction is important because when p(success|cond) for some cond is low, its variance is also very small, but prop.test(39,67) doesn't reflect this. That could be 67 successes out of a small sample (and high chance of success), or out of a huge sample (and low chance of success). Edit: whoops, I didn't notice other comments point out this issue