4 ms·

Don't just run tests longer - run tests for a pre-defined amount of time instead of "until you see a result you like".

scuba_man_spiff 11y ago
Your comment hits the nail on the head here. Standard statistical tests used in a/b testing are based on one check. If someone is checking repeatedly on a test until they get a 'significant' result, your chance of getting a getting a false positive is many X the stated significance. Best practice - set a pre-defined end, and one or two defined early check-in points where only make an early call if result is overwhelmingly significant or if the business has fallen off a cliff.
- stdbrouw 11y ago
  That would be best practice if you insist on using null hypothesis significance testing and only wanted to use classical frequentist statistics. We really can do much better these days with multi-armed bandits, and by focusing on effect sizes and credible intervals rather than a yes/no answer to a hypothesis.
  - jacalata 11y ago
    Sounds like I need to update my stats knowledge! Do you happen to know of a good place to start learning about today's state of the art?
yummyfajitas 11y ago
This is necessary with textbook statistical methods which are explicitly based on the assumption of looking at exactly one time. However, modern tools (Optimizely, VWO, Qubit) do not require this. [1] If you use one of these tools it's completely safe to run tests until the test says win. There are still caveats to account for the real world - e.g., only stop the test after an integer number of weeks - but statistically this is a solved problem. [1] The blog post under discussion was written before Optimizely's StatsEngine and VWO's SmartStats.
anarchitect 11y ago
Exactly. And for whole business cycles (for us, that's weeks) to accommodate changes in behaviour and traffic throughout the cycle. We try to run as many experiments as our traffic can handle, but we always estimate the sample size upfront when planning the test.

Your comment hits the nail on the head here. Standard statistical tests used in a/b testing are based on one check. If someone is checking repeatedly on a test until they get a 'significant' result, your chance of getting a getting a false positive is many X the stated significance. Best practice - set a pre-defined end, and one or two defined early check-in points where only make an early call if result is overwhelmingly significant or if the business has fallen off a cliff.

stdbrouw 11y ago
That would be best practice if you insist on using null hypothesis significance testing and only wanted to use classical frequentist statistics. We really can do much better these days with multi-armed bandits, and by focusing on effect sizes and credible intervals rather than a yes/no answer to a hypothesis.
- jacalata 11y ago
  Sounds like I need to update my stats knowledge! Do you happen to know of a good place to start learning about today's state of the art?

yummyfajitas 11y ago

This is necessary with textbook statistical methods which are explicitly based on the assumption of looking at exactly one time. However, modern tools (Optimizely, VWO, Qubit) do not require this. [1] If you use one of these tools it's completely safe to run tests until the test says win. There are still caveats to account for the real world - e.g., only stop the test after an integer number of weeks - but statistically this is a solved problem. [1] The blog post under discussion was written before Optimizely's StatsEngine and VWO's SmartStats.

anarchitect 11y ago

Exactly. And for whole business cycles (for us, that's weeks) to accommodate changes in behaviour and traffic throughout the cycle. We try to run as many experiments as our traffic can handle, but we always estimate the sample size upfront when planning the test.