3 ms·

The tool has that built-in. As soon as significance is achieved, we show a "winner", so to answer your question, instead talking about statistical significance

The tool has that built-in. As soon as significance is achieved, we show a "winner", so to answer your question, instead talking about statistical significance we focus on the more positive aspect of an A/B test. Those not initiated in Statistics find this a little difficult to understand. I personally go about it like this: Me: "If you flip a coin 10 times and it shows heads 7 times, can you say with certainty that the coin is loaded?" Them: "Umm, no not really" Me: "Ok, what if I flip it 100 times and it shows head 70 times?" Them: "Yeah maybe" Me : "What if 7000 out of 10,000?" Them: "Of course it's loaded" And that's how I explain the basic concept of statistical significance, but without the math. More data means it becomes more 'obvious'. To take it a notch further, I ask "If your life depended on it, would you still bet that the coin is loaded?" People usually become unsure after this and I joke that their previous decision was based on 95% significance, but if they really want to be sure, they should wait for 99% significance.

Ovid 13y ago
As soon as significance is achieved, we show a "winner" If you've described that accurately, you're hurting some of your customers. It's not well-known (though it should be obvious, in retrospect), but a test can produce a false positive when, in fact, it's a negative. The problem is that there is always noise in customer data and sometimes you'll randomly have enough conversions such that one of your tests will show a winner when it's not. Matts Einersen has some simulation code to show this: http://www.einarsen.no/is-your-ab-testing-effort-just-chasing-statistical-ghosts/ http://www.einarsen.no/is-your-ab-testing-effort-just-chasin... Evan Miller has a more in-depth (but harder to follow) discussion here: http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html If, in fact, you show the winner as soon as significance is achieved, it's then trivial to demonstrate that some of your customers will lose money as a result. However, if you take this knowledge and then find a good way to incorporate this into your tool, you may have another great selling point that competitors do not (that being said, the idea is complicated enough that some customers won't understand it).
- siddharthdeswal 13y ago
  Trying to do exactly that at http://visualwebsiteoptimizer.com/ab-split-test-duration/ http://visualwebsiteoptimizer.com/ab-split-test-duration/ and http://visualwebsiteoptimizer.com/ab-split-significance-calculator/ http://visualwebsiteoptimizer.com/ab-split-significance-calc... but yeah, we need a better way to present this.
  - Ovid 13y ago
    Have you considered adding a "recommend days" field to your product? Given the number of people per experiment and the number of conversions, you could potentially calculate a minimum time to run the experiment and simply explain that sometimes you are more likely to get false positives prior to the minimum time? Also, it makes me very happy to know that you're aware of this problem and are trying to tackle it. It makes me more likely to recommend it to clients.
IanCal 13y ago
Nice. I'm impressed with your pages telling people how long to run their experiments for as well. I think early stopping can be a problem, but there's a difficult gap there. Good experiments are difficult to set up, and may be too much for a site owner who just wants to know if he should make the damn button green or red. The more you force someone to set up a complex trial, the more they'll want to go to your competitors who give them a nice flashy message saying "GREEN BUTTON WINS BY 1.4%" without all the hassle. It looks like you've got a good balance there :)

Ovid 13y ago

As soon as significance is achieved, we show a "winner" If you've described that accurately, you're hurting some of your customers. It's not well-known (though it should be obvious, in retrospect), but a test can produce a false positive when, in fact, it's a negative. The problem is that there is always noise in customer data and sometimes you'll randomly have enough conversions such that one of your tests will show a winner when it's not. Matts Einersen has some simulation code to show this: http://www.einarsen.no/is-your-ab-testing-effort-just-chasing-statistical-ghosts/ http://www.einarsen.no/is-your-ab-testing-effort-just-chasin... Evan Miller has a more in-depth (but harder to follow) discussion here: http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html If, in fact, you show the winner as soon as significance is achieved, it's then trivial to demonstrate that some of your customers will lose money as a result. However, if you take this knowledge and then find a good way to incorporate this into your tool, you may have another great selling point that competitors do not (that being said, the idea is complicated enough that some customers won't understand it).

siddharthdeswal 13y ago
Trying to do exactly that at http://visualwebsiteoptimizer.com/ab-split-test-duration/ http://visualwebsiteoptimizer.com/ab-split-test-duration/ and http://visualwebsiteoptimizer.com/ab-split-significance-calculator/ http://visualwebsiteoptimizer.com/ab-split-significance-calc... but yeah, we need a better way to present this.
- Ovid 13y ago
  Have you considered adding a "recommend days" field to your product? Given the number of people per experiment and the number of conversions, you could potentially calculate a minimum time to run the experiment and simply explain that sometimes you are more likely to get false positives prior to the minimum time? Also, it makes me very happy to know that you're aware of this problem and are trying to tackle it. It makes me more likely to recommend it to clients.

IanCal 13y ago

Nice. I'm impressed with your pages telling people how long to run their experiments for as well. I think early stopping can be a problem, but there's a difficult gap there. Good experiments are difficult to set up, and may be too much for a site owner who just wants to know if he should make the damn button green or red. The more you force someone to set up a complex trial, the more they'll want to go to your competitors who give them a nice flashy message saying "GREEN BUTTON WINS BY 1.4%" without all the hassle. It looks like you've got a good balance there :)