3 ms·
There are a number of valid ways to run and analyze A/B tests. Bayesian approaches look rather different than what you're describing.
by btilly 14y ago
There are a number of valid ways to run and analyze A/B tests. Bayesian approaches look rather different than what you're describing.
- jfarmer 14y agoThe original commenter was using off-the-shelf A/B testing software, so the odds of it doing anything other than a simple t-test are virtually zero. Not sure that the frequentist vs. Bayesean debate is the most relevant thing for him right now. I felt it best to leave out nuance that didn't help him understand why his software was showing a statistically significant outcome for an A/A test.
- robrenaud 14y agoYou are assuming the framework isn't broken or misconfigured.
- jfarmer 14y agoYou're not going to out-pedant me, damnit! :P Given the original comment, yes, I think it's more likely he just didn't understand why a A/A test might sometimes show a false positive. Even if the system were misconfigured, there's no reason to think it would manifest as a false positive in an A/A test. There are lots of ways it could manifest.