4 ms·
Rather than using the modulo on some user id, you'd generally want to use the modulo on md5(test_name + user_id). You can't be sure that user ids are evenly di
by patio11 17y ago
Rather than using the modulo on some user id, you'd generally want to use the modulo on md5(test_name + user_id). You can't be sure that user ids are evenly distributed -- for example, if they're monotonically increasing, then there might be subtle interactions between signup order and your A vs. B variations which could poison the validity of your results.
Bonus points: if you modulo on the user ID then Bob will always get whatever alternative you write first, which could be suboptimal unless you write your code in random order (which you don't). If you modulo the above MD5 thing, Bob will sometimes get A and sometimes get B for different tests, but always get consistent results within the same test.
- btilly 17y agoYou recommended the same thing at http://news.ycombinator.com/item?id=812421 http://news.ycombinator.com/item?id=812421. My question to you then was what you'd do if you started with 4 versions then proved that one was bad. Your response was to start over. If you instead keep a table of which people are in which version, progressively removing versions while not throwing away existing data is easy to do. This also allows you to set up tests where A and B get unequal splits of the data. The drawback is that you need a new table that grows fast and is heavily accessed. However I've found that cost to be quite reasonable.
- patio11 17y agoI think we're largely on the same page vis-a-vis the performance vs. ease of mutating tests tradeoff. In my personal workflow, tests are cheap, so throwing them away doesn't really bother me. The performance hit bothers me a little, as it would potentially cause problems for some people using my A/B testing library. If your performance needs tolerate a read on every render of an A/B tested element and a write for every unique visitor seeing an A/B tested element, though, mazel tov, your way should work swimmingly. For folks interested in weighting without the database: you can do unequal weights with the modulo method, too. Imagine if you had five choices as follows: [A, A, B, B, B]. This array can get computed once from an arbitrarily simple (or complex) representation in your program, and then cached somewhere where accessing it is very fast. (I stick it in Memcached.) Thus you get all the weights with none of the calories. You don't get trivial re-weighting, though.
- btilly 17y agoThe problem isn't how expensive tests are, it is that someone is waiting for the test answer. Throwing away several days of data causes them to wait longer, which makes for unhappy product managers.