leo_pekelis

6 ms·

1.
Approximate Counting and Statistical Significance: 2 Ideas That Don’t Play Nice(medium.com)
3 points by leo_pekelis 10y ago|0 comments
2.
by leo_pekelis 10y ago
Hi, this is Leo, Optimizely's statistician. If you're looking for a more scientific paper, maybe take a look at this one we wrote recently: http://arxiv.org/abs/1512.04922 Should have everything you would eve
3.
by leo_pekelis 12y ago
Hey, If you are interested in learning more about the problems we solved, we lay them out in much more detail in our blog post. It’s a meaty topic and so the post is not short. http://blog.optimizely.com/2015/01/20
4.
by leo_pekelis 12y ago
While I do agree that a sequential hypothesis test like the one we implemented in Stats Engine is different than a completely Bayesian method, I wouldn’t necessarily call it less powerful. In fact, numerous optimality properties exist showi
5.
by leo_pekelis 12y ago
I do agree with you that with sequential testing it is possible to get much slower results. This is actually similar to using a sample size calculator for a classical t-test. If you set your minimum detectible effect (MDE) much smaller than
6.
by leo_pekelis 12y ago
Sorry for the delay in responding. HN thinks we are responding too fast to comments over here. =)
7.
by leo_pekelis 12y ago
One point of clarification, the y-axis on the chart does have the same meaning for both lines. It is 1 minus the chance of committing a type I error. I think you do point out an important nuance that under sequential testing a type I error
8.
by leo_pekelis 12y ago
Hello, Leo, Optimizely's in-house statistician here. The graph you reference is a schematic to show the differences between Optimizely’s previous statistical platform and Stats Engine. It shows a monotone non-decreasing significance be