14 ms·

Wait, nothing is wrong with collecting data until you see something significant. In fact, that's the better way to do things. You just need to make sure your ma

Wait, nothing is wrong with collecting data until you see something significant. In fact, that's the better way to do things. You just need to make sure your math acknowledges the procedure.

mehwoot 13y ago
In practice, doing this will lead to what the author talks about under "Stopping tests as soon as you see winning results will create false positives." It doesn't make your data invalid, but it will generally lead to poor methodology if you are not careful.
joosters 13y ago
http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html gives a good explanation about how this can ruin your tests.
- darkxanthos 13y ago
  He even says this is possible via Bayesian experiment design: FTA: Bayesian experiment design: With Bayesian experiment design you can stop your experiment at any time and make perfectly valid inferences.
zorpner 13y ago
> You just need to make sure your math acknowledges the procedure. I'd love to hear some clarification on how "[making] sure your math acknowledges" stopping randomized tests early works, because it flies in a face of how actual statistics works.
- gwern 13y ago
  > it flies in a face of how actual statistics works. The terms you're looking for are 'sequential testing', 'optimal stopping', and 'adaptive trials'.
- xenophanes 13y ago
  consider fair coin flip sequences. stopping at arbitrary points of your choice can never really affect anything serious. but you need to make sure you're doing math right, meaning if you stop when something gets a lead you don't use incorrect math that says the coin is biased. data is data. as long as the method being used to collect individual data points is fine, and they are collected independently, then the data you get as a result is gonna be OK, the rest (like arbitrary stop time) doesn't ruin it. you just have to avoid bad math. what ruins data is stuff like throwing 10% of the heads results in the trash or using other approaches in which data can be selectively discarded or not discarded. so just stopping arbitrarily can be a problem if you might never stop and throw out the results if you don't like them. but if you do something like "stop after 1 million data points max, or when i feel like it earlier" then your data is still OK because it cannot get selectively ignored. stopping earlier cannot make a fair coin look unfair or anything like that. this is not some random unknown position that flies in the face of how actual statistics works. something like this is the standard bayesian position, and i think it's true. (i strongly object to bayesian epistemology, but i think bayesian statistics is correct). not ALL stopping rules are OK but lots are. you don't HAVE to use simple ones like "gather X data points, stop". see stuff like: http://andrewgelman.com/2014/02/13/stopping-rules-bayesian-analysis/ http://andrewgelman.com/2014/02/13/stopping-rules-bayesian-a... http://lesswrong.com/lw/mt/beautiful_probability/ http://lesswrong.com/lw/mt/beautiful_probability/ > And then there's the Bayesian reply: "Excuse you? The evidential impact of a fixed experimental method, producing the same data, depends on the researcher's private thoughts? And you have the nerve to accuse us of being 'too subjective'?"
  - xenophanes 13y ago
    Previous HN discussion: https://news.ycombinator.com/item?id=3188816 https://news.ycombinator.com/item?id=3188816
beagle3 13y ago
By "acknowledge" you probably mean apply an SPRT (serial probability ratio test). The Neymann-Pearson lemma is also relevant.
hvidgaard 13y ago
Setting a 90% power target is a way to decide when you have something significant.

Wait, nothing is wrong with collecting data until you see something significant. In fact, that's the better way to do things. You just need to make sure your math acknowledges the procedure.

mehwoot 13y ago

In practice, doing this will lead to what the author talks about under "Stopping tests as soon as you see winning results will create false positives." It doesn't make your data invalid, but it will generally lead to poor methodology if you are not careful.

joosters 13y ago

http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html gives a good explanation about how this can ruin your tests.

darkxanthos 13y ago
He even says this is possible via Bayesian experiment design: FTA: Bayesian experiment design: With Bayesian experiment design you can stop your experiment at any time and make perfectly valid inferences.

zorpner 13y ago

> You just need to make sure your math acknowledges the procedure. I'd love to hear some clarification on how "[making] sure your math acknowledges" stopping randomized tests early works, because it flies in a face of how actual statistics works.

gwern 13y ago
> it flies in a face of how actual statistics works. The terms you're looking for are 'sequential testing', 'optimal stopping', and 'adaptive trials'.
xenophanes 13y ago
consider fair coin flip sequences. stopping at arbitrary points of your choice can never really affect anything serious. but you need to make sure you're doing math right, meaning if you stop when something gets a lead you don't use incorrect math that says the coin is biased. data is data. as long as the method being used to collect individual data points is fine, and they are collected independently, then the data you get as a result is gonna be OK, the rest (like arbitrary stop time) doesn't ruin it. you just have to avoid bad math. what ruins data is stuff like throwing 10% of the heads results in the trash or using other approaches in which data can be selectively discarded or not discarded. so just stopping arbitrarily can be a problem if you might never stop and throw out the results if you don't like them. but if you do something like "stop after 1 million data points max, or when i feel like it earlier" then your data is still OK because it cannot get selectively ignored. stopping earlier cannot make a fair coin look unfair or anything like that. this is not some random unknown position that flies in the face of how actual statistics works. something like this is the standard bayesian position, and i think it's true. (i strongly object to bayesian epistemology, but i think bayesian statistics is correct). not ALL stopping rules are OK but lots are. you don't HAVE to use simple ones like "gather X data points, stop". see stuff like: http://andrewgelman.com/2014/02/13/stopping-rules-bayesian-analysis/ http://andrewgelman.com/2014/02/13/stopping-rules-bayesian-a... http://lesswrong.com/lw/mt/beautiful_probability/ http://lesswrong.com/lw/mt/beautiful_probability/ > And then there's the Bayesian reply: "Excuse you? The evidential impact of a fixed experimental method, producing the same data, depends on the researcher's private thoughts? And you have the nerve to accuse us of being 'too subjective'?"
- xenophanes 13y ago
  Previous HN discussion: https://news.ycombinator.com/item?id=3188816 https://news.ycombinator.com/item?id=3188816

beagle3 13y ago

By "acknowledge" you probably mean apply an SPRT (serial probability ratio test). The Neymann-Pearson lemma is also relevant.

hvidgaard 13y ago

Setting a 90% power target is a way to decide when you have something significant.