4 ms·
"We left the test running until the results were statistically significant to a 99% confidence level." This is absolutely the wrong thing to do - not that I'm
by Jabbles 16y ago
"We left the test running until the results were statistically significant to a 99% confidence level."
This is absolutely the wrong thing to do - not that I'm doubting the conclusion, but the data does not support that confidence level.
http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html
- jarin 16y agoThis is why I'm terrible at statistics. To me, it looks almost like wave function collapse. The results are good only if you do a set number of observations (say 500) instead of waiting for a significant result (say it happens at 623). But what if you had decided to run 623 tests at the beginning?
- Jabbles 16y agoNo problem with that. But compare these two experiments: for i in range 623: data.add_result() s = calculate_significance(data) if s > 0.95: publish() for i in range 623: data.add_result() s = calculate_significance(data) if s > 0.95: publish() break The second one gives you many more chances to succeed, which must result in your confidence in the answer going down.