6 ms·
I do this professionally as my sole job. This is one of the very few papers I've read that seem completely legit to me. I especially love their point on necessa
by darkxanthos 13y ago
I do this professionally as my sole job. This is one of the very few papers I've read that seem completely legit to me. I especially love their point on necessary sample sizes to get to a 90% power.
- ep103 13y agoHow do you calculate the correct sample size for a test, to achieve the correct "power"?
- carbocation 13y agoHere are a few resources: http://statpages.org/#Power http://statpages.org/#Power
- gwern 13y agoFor binomial scenarios like a stock A/B test, most statistical environments will have some sort of built-in power functions. For example, R does; an example: http://www.gwern.net/AB%20testing#power-analysis http://www.gwern.net/AB%20testing#power-analysis
- tel 13y agoFor simple tests you can reverse the mathematics to get good estimates of how many observations are needed given a goal for your desired power and tolerance for false positives. Asking for greater power makes your test more sensitive ("buys a bigger telescope") at the cost of increased sample size. Asking for fewer false positives ("cleaning the lenses") costs similarly. For more sophisticated tests, ones less likely to be seen in an A/B scenario, you might not be able to reverse the mathematics and get a direct answer, so often people will run simulation studies to guess at the needed sample size.
- jamiequint 13y agoLehr's Equation (Rule of 16) is generally a good estimator. It is explained (but not referenced) in a similar article here: http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html
- darkxanthos 13y agoI use simulations so I can avoid math and analyze the power of any distribution I can model in a consistent manner. I wrote something about it here: http://www.databozo.com/2013/10/12/Finding_a_sample_size_for_your_split_test.html http://www.databozo.com/2013/10/12/Finding_a_sample_size_for... EDIT: typo
- krallja 13y agoWhy is 90% power the magic number? What's wrong with 89.99%? Or 99.99%?
- dfbrown 13y agoIt's not, you can use whatever power you want.
- al2o3cr 13y agoNothing's "wrong" with 89.99% - 90% is simply a preferred value. The point of the exercise is not to pick a magic number, but to pick A number and therefore a target sample size before collecting results. Without something like this, it's very easy to fall into the "we'll collect data until something that looks significant appears" trap.
- tedsanders 13y agoWait, nothing is wrong with collecting data until you see something significant. In fact, that's the better way to do things. You just need to make sure your math acknowledges the procedure.
- mehwoot 13y agoIn practice, doing this will lead to what the author talks about under "Stopping tests as soon as you see winning results will create false positives." It doesn't make your data invalid, but it will generally lead to poor methodology if you are not careful.
- joosters 13y agohttp://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html gives a good explanation about how this can ruin your tests.
- darkxanthos 13y agoHe even says this is possible via Bayesian experiment design: FTA: Bayesian experiment design: With Bayesian experiment design you can stop your experiment at any time and make perfectly valid inferences.