3 ms·
The Importance of Sequential Testing
- noelwelsh 13y ago[I'm assuming the author posted this. MidsizeBlowfish has no profile information.] Good stuff, and just one of the many reasons that most A/B testing practice is flawed. I haven't seen the SPRT used much in practice. I'm not sure why. A quick Googling indicates non-parametric sequential tests have been developed. What about the case with more than two alternatives? My field is bandit algorithms, which are a generalisation of hypothesis tests. Here there are algorithms like Empirical Bernstein Stopping (http://icml2008.cs.helsinki.fi/papers/523.pdf http://icml2008.cs.helsinki.fi/papers/523.pdf) which achieve the same early stopping from a different approach. If this is of interest sign up to the course I'm running (http://noelwelsh.com/data/2013/08/16/free-bandit-course-draft-outline/ http://noelwelsh.com/data/2013/08/16/free-bandit-course-draf...) -- I plan to cover this algorithm soon. BTW, you have a very formal writing style. I think if you're trying for a more general audience you should loosen up a bit. This ain't a journal paper. :-)
- MidsizeBlowfish 13y agoHaha, yes, I am the author. I do have a very formal style, an unfortunate product of too many years in grad school. The SPRT is a nice toy test, but only useful for point hypotheses. There are generalizations to more realistic composite hypotheses, though.
- noelwelsh 13y agoIt would be interesting to me if you covered the generalizations. This field seems to have evolved entirely separately from the banditry I'm accustomed to.
- MidsizeBlowfish 13y agoI'll put a follow up on some other sequential tests into my list of blog-posts-to-do and let you know when it happens.
- nitrogen 13y agoA heads up to the author: with narrower browser window sizes (e.g. less than ~1024px), the left side menu overlaps the text.
- MidsizeBlowfish 13y agoJust noticed that on my iPad, will look into it.