3 ms·
Did you know that your rule of "do the tests until significance is reached" denies the use of chain probability and most tests based on it? You can show that i
by AstralStorm 11y ago
Did you know that your rule of "do the tests until significance is reached" denies the use of chain probability and most tests based on it?
You can show that in the limit, your rule will always reach any given threshold just by pure luck.
You really need to set the number of trials or samples before performing them.
Here's a more in depth explanation:
http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html
- btilly 11y agoThe procedure I suggested starts with setting the maximum number of conversions before beginning the test. It is emphatically not the procedure that Evan Miller criticizes. In fact see http://www.evanmiller.org/sequential-ab-testing.html http://www.evanmiller.org/sequential-ab-testing.html for Evan Miller suggesting a similar procedure to mine, based on past conversations that I have had with him. The difference between that one and mine is that I focus on "make the best decision available, even though it may be only little better than a coin flip" while he tries to only decide when you can decide with confidence. In past conversations we have agreed that each other's procedures are reasonable, but they achieve different goals.