3 ms·
Agreed. The point of most A/B testing is not to find the best approach per se, but have the greatest number of successes. If failures essential don't matter (e
by paulddraper 8y ago
Agreed. The point of most A/B testing is not to find the best approach per se, but have the greatest number of successes.
If failures essential don't matter (e.g. number of bacteria killed in a petri dish), sure use frequentist p-value.
If failure do matter (e.g. number of patients killed), use Bayesian multi-armed bandit.
Wrote a blog post on it: https://www.lucidchart.com/blog/the-fatal-flaw-of-ab-tests-peeking https://www.lucidchart.com/blog/the-fatal-flaw-of-ab-tests-p...