3 ms·

That's not how you use prop.test. What you've tested using that invocation is the null hypothesis that the underlying probability of 39/67 is 0.5. If you want

That's not how you use prop.test. What you've tested using that invocation is the null hypothesis that the underlying probability of 39/67 is 0.5. If you want to perform a test of a difference of two proportions, you need to do: prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions)) I don't have experience with A/B testing, so I'm not sure if this is typically or best handled using this particular statistical test. Edit: The first parameter should be c(39, 28), meaning the total conversions in each group. I have no excuse beyond being tired. Edit 2: To clarify, I think he should still use the two-sample form of prop.test, especially since we did not know at the time of his posting that the sample sizes are equal.

no_gravity 8y ago
What you've tested using that invocation is the null hypothesis that the underlying probability of 39/67 is 0.5. Isn't that equivalent to my interpretation of the test result? "In a world where it makes no difference which design is used, you would get a result as significant as this 22% of the time". If you want to perform a test of a difference of two proportions, you need to do: prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions)) Do you mean c(39,28)? Because group_a had 39 hits and group_b hat 28. Doing so with the group sizes Bemmu stated (3000/3000) also gives me a p value of 0.22. As long as the group sizes are equal, the test is not very sensitive to the sizes.
- muraiki 8y ago
  I think there is a difference in the approaches, given that the Chi-squared test statistic for the two-sample version is ~1.52, while for your one-sample version it is ~1.81. If group size doesn't matter and if you're justified in adding up successes as you have, I'd expect the test statistics to be nearly the same. Edit: I'd expect them to be nearly the same since the Chi-squared distributions would be parameterized similarly, so if we have similar results, we should see similar test statistics. Maybe my reasoning here is incorrect though!

no_gravity 8y ago

What you've tested using that invocation is the null hypothesis that the underlying probability of 39/67 is 0.5. Isn't that equivalent to my interpretation of the test result? "In a world where it makes no difference which design is used, you would get a result as significant as this 22% of the time". If you want to perform a test of a difference of two proportions, you need to do: prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions)) Do you mean c(39,28)? Because group_a had 39 hits and group_b hat 28. Doing so with the group sizes Bemmu stated (3000/3000) also gives me a p value of 0.22. As long as the group sizes are equal, the test is not very sensitive to the sizes.

muraiki 8y ago
I think there is a difference in the approaches, given that the Chi-squared test statistic for the two-sample version is ~1.52, while for your one-sample version it is ~1.81. If group size doesn't matter and if you're justified in adding up successes as you have, I'd expect the test statistics to be nearly the same. Edit: I'd expect them to be nearly the same since the Chi-squared distributions would be parameterized similarly, so if we have similar results, we should see similar test statistics. Maybe my reasoning here is incorrect though!