4 ms·
Thanks for this great article. Both this article and Effective A/B Testing (http://elem.com/~btilly/effective-ab-testing/ http://elem.com/~btilly/effective-ab-
by mef 15y ago
Thanks for this great article.
Both this article and Effective A/B Testing (http://elem.com/~btilly/effective-ab-testing/ http://elem.com/~btilly/effective-ab-testing/ slide 51) say changing the proportions mid-test is bad. Can anyone elaborate on why this might be?
For example, if I'm randomly showing my existing landing page or a new landing page to visitors at a 2/3-1/3 split and measuring CTR to the signup page, and I change the proportion to 1/2-1/2, why would that skew results?
- tel 15y agoI think it depends somewhat on what the proportions you're talking about are. Ben Tilly's slide seems to be saying that changing already enrolled users from A to B is bad, clearly. I don't think changing the filtering rate on new users would be bad, but I haven't looked at the mechanics in a while.
- Estragon 15y agoI'm curious about this too. The individual tests should be identifiable regardless of the proportions in which they were assigned.