4 ms·

I assume people who have run these "no results" A/B tests only gave up after enough traffic to establish no significant result was found given pre-determined X

I assume people who have run these "no results" A/B tests only gave up after enough traffic to establish no significant result was found given pre-determined X power and Y statistical significance cutoff. We have a calculator here to help determine what is the maximum number of visitors you need before giving up on a test: http://visualwebsiteoptimizer.com/ab-split-test-duration/ http://visualwebsiteoptimizer.com/ab-split-test-duration/

noelwelsh 16y ago
That's a pretty big assumption to make. I've seen people doing A/B testing w/ a few thousand hits max, which is not enough for most scenarios.
- paraschopra 16y ago
  What's the big assumption? We are using math here, not heuristics. There is no thumb rule like thousands and hundreds of visitors needed to get significant results.
  - noelwelsh 16y ago
    I was questioning this assumption of yours: "I assume people who have run these "no results" A/B tests only gave up after enough traffic to establish no significant result was found given pre-determined X power and Y statistical significance cutoff". I am familiar with the maths for A/B testing. The figure of approximately 100'000 hits comes from "Controlled experiments on the web: survey and practical guide". As I stated, this assumes 5% conversion rate and a few other things. Here's the quote: "If, however, you were only looking for 5% change in conversion rate (not revenue), a lower variability OEC [Overall Evaluation Criteria] based on point 3.b can be used. Purchase, a conversion event, is modeled as a Bernoulli trial with p = 0.05 being the probability of a purchase. The standard deviation of a Bernoulli is √p(1− p) and thus you will need less than 122,000 users to achieve the desired power based on 16 ∗ (0.05 · (1−0.05))/(0.05 · 0.05)^2." (The actual value is 121'600.)
    - ltjohnson 16y ago
      122,000 seemed absurdly large to me so I just looked up the paper you reference [1]. It looks like there is a calculation error in the paper. The formula they use is n = 16 σ^2 / Δ^2. σ is the standard deviation, and Δ is the size of the difference. Thus in this problem, Δ = 0.05 and their formula gives n = 16 * (0.05 * (1-0.05)) / 0.05^2 = 304. This is much more in line with what you get from using a 2 sample proportion test (with H_a: p_1 =/= p_2), ~440 in each group [2]. But maybe I misunderstand their formula. [1] http://exp-platform.com/hippo_long.aspx http://exp-platform.com/hippo_long.aspx [2] http://statpages.org/proppowr.html http://statpages.org/proppowr.html edit: fixed Greek letters and added final comment.
      - noelwelsh 16y ago
        Interesting -- I'll check it out. (My working day has almost finished, so this thread will probably be dead before I get back it.)
      - gjm11 16y ago
        I think you misunderstand what they're doing with the formula. Delta is the size of the difference you want to detect. In this case, they're saying they want to detect a 5% (relative) change in a 5% (absolute) conversion rate. So σ^2 = 0.05 . (1-0.05), but Δ is 5% of 5% or 0.0025 and the denominator needs to be 0.0025^2. If you go to your reference [2] and enter the numbers 0.05, 80, 0.05, 0.0525, 1.0, you'll see that they come up with a sample size of about 122k in each group (so 244k in both together). (The figure of 304 or 440 is what you would get if you wanted to detect an absolute change of 5% in the conversion rate: going from 5% to 0% or to 10%.)
        ltjohnson 16y ago
        You're right, reading the paper that way, they are trying to detect a change in convergence rate from 5% to 5.25%, I was confused by the use of % in two separate contexts in the same sentence. That being said, I think this is not a good argument against A/B testing. Fair enough, it would take a very large sample (122K is close enough) to detect a change from 5% to 5.25%. Being concerned about a change that small seems really silly unless 0.0025 * N visitors * revenue per user is a big enough number to be concerned with. I contend it won't be unless either (1) N visitors is very large or (2) revenue per user is very large. If (1) is true, then testing on 122K users is not a big deal. If (2) is true you probably want to have a much more targeted approach, like someone doing sales.

noelwelsh 16y ago

That's a pretty big assumption to make. I've seen people doing A/B testing w/ a few thousand hits max, which is not enough for most scenarios.

paraschopra 16y ago
What's the big assumption? We are using math here, not heuristics. There is no thumb rule like thousands and hundreds of visitors needed to get significant results.
- noelwelsh 16y ago
  I was questioning this assumption of yours: "I assume people who have run these "no results" A/B tests only gave up after enough traffic to establish no significant result was found given pre-determined X power and Y statistical significance cutoff". I am familiar with the maths for A/B testing. The figure of approximately 100'000 hits comes from "Controlled experiments on the web: survey and practical guide". As I stated, this assumes 5% conversion rate and a few other things. Here's the quote: "If, however, you were only looking for 5% change in conversion rate (not revenue), a lower variability OEC [Overall Evaluation Criteria] based on point 3.b can be used. Purchase, a conversion event, is modeled as a Bernoulli trial with p = 0.05 being the probability of a purchase. The standard deviation of a Bernoulli is √p(1− p) and thus you will need less than 122,000 users to achieve the desired power based on 16 ∗ (0.05 · (1−0.05))/(0.05 · 0.05)^2." (The actual value is 121'600.)
  - ltjohnson 16y ago
    122,000 seemed absurdly large to me so I just looked up the paper you reference [1]. It looks like there is a calculation error in the paper. The formula they use is n = 16 σ^2 / Δ^2. σ is the standard deviation, and Δ is the size of the difference. Thus in this problem, Δ = 0.05 and their formula gives n = 16 * (0.05 * (1-0.05)) / 0.05^2 = 304. This is much more in line with what you get from using a 2 sample proportion test (with H_a: p_1 =/= p_2), ~440 in each group [2]. But maybe I misunderstand their formula. [1] http://exp-platform.com/hippo_long.aspx http://exp-platform.com/hippo_long.aspx [2] http://statpages.org/proppowr.html http://statpages.org/proppowr.html edit: fixed Greek letters and added final comment.
    - noelwelsh 16y ago
      Interesting -- I'll check it out. (My working day has almost finished, so this thread will probably be dead before I get back it.)
    - gjm11 16y ago
      I think you misunderstand what they're doing with the formula. Delta is the size of the difference you want to detect. In this case, they're saying they want to detect a 5% (relative) change in a 5% (absolute) conversion rate. So σ^2 = 0.05 . (1-0.05), but Δ is 5% of 5% or 0.0025 and the denominator needs to be 0.0025^2. If you go to your reference [2] and enter the numbers 0.05, 80, 0.05, 0.0525, 1.0, you'll see that they come up with a sample size of about 122k in each group (so 244k in both together). (The figure of 304 or 440 is what you would get if you wanted to detect an absolute change of 5% in the conversion rate: going from 5% to 0% or to 10%.)
      - ltjohnson 16y ago
        You're right, reading the paper that way, they are trying to detect a change in convergence rate from 5% to 5.25%, I was confused by the use of % in two separate contexts in the same sentence. That being said, I think this is not a good argument against A/B testing. Fair enough, it would take a very large sample (122K is close enough) to detect a change from 5% to 5.25%. Being concerned about a change that small seems really silly unless 0.0025 * N visitors * revenue per user is a big enough number to be concerned with. I contend it won't be unless either (1) N visitors is very large or (2) revenue per user is very large. If (1) is true, then testing on 122K users is not a big deal. If (2) is true you probably want to have a much more targeted approach, like someone doing sales.