3 ms·
Hashing has one major advantage over random assignment: reproducibility. If you have some way of IDing Mary Smith, you can consistently serve her the same test
by patio11 16y ago
Hashing has one major advantage over random assignment: reproducibility. If you have some way of IDing Mary Smith, you can consistently serve her the same tests on all machines, and expose the tests she is participating in to bug investigators, without having to actually store a users => chosen test alternatives map anywhere. Those can get fairly sizable and the access patterns suck.
A/Bingo does it by hashing taking MD5(user_id . test_name). You have to store user_I'd, but you're doing that anyhow.
- btilly 16y agoIf you do random assignment, and save the random assignments somewhere, then you can also get reproducibility. If you do random assignment, and don't save the random assignments anywhere, then you have no idea how many people were in your test versions. Which is not a good idea. (You can estimate this data. I've done it. But doing it properly is surprisingly tricky. It is very, very easy to do it wrong.) There are several benefits to this approach. The first benefit is that if you're testing a particular page, you can easily make your test only include people who have hit that page. This will cause results to converge more quickly than if you don't know which people on your site actually hit that page. The second, and sometimes critical, benefit is that you know exactly when someone entered the test. Multiple times when testing things with a longer sales cycle I've encountered the situation where a particular test version causes the sales cycle to become compressed, but may or may not provide a long-term improvement in conversion. Access to data about when people entered your test allows you to examine A/B test results only for people who have been in the test long enough to be likely to have completed either version. A third benefit is that if you're testing multiple versions, then you can just continue the test and drop poorly performing versions as you prove that they are suboptimal. The downside, as you say, is that the map of who is in what version can get very large. In my experience, though, the access patterns are not that bad. Particularly not if you are already using sessions, and can cache that information in the current session, so that most page hits don't have to fetch the A/B test version. Furthermore this is not data that you need to join to anything else on your live website. Therefore it is a perfect candidate to move somewhere like Redis.