4 ms·
This is an implementation issue. Assuming you are talking about presenting users inconsistent interfaces in terms of User Bob seeing both the A and B, then thi
by patio11 17y ago
This is an implementation issue.
Assuming you are talking about presenting users inconsistent interfaces in terms of User Bob seeing both the A and B, then this is not too hard to solve. If your issue is with evolving your interface rapidly enough that User Bob would see A1, B2, A3, A4, and A5 in the period of two weeks, then that's a bit outside the scope of this discussion, but it is a pace-of-change issue rather than strictly speaking an A/B testing issue.
Anyhow, back to Bob. When we first see Bob on our site, we assign him a random identifier which is presumed to be unique. Should Bob log in or otherwise demonstrate his identity to us, we associate the random identifier durably with the identity. Any other anonymous user who signs in as Bob will henceforth have their random identifier overwritten with the canonical Bob identifier.
Observe that this means anyone who we know is Bob will always have the same constant random identifier, from here on out.
OK, now for each A/B test we do, we take a unique identifier of the test (the database primary key, or the name if we can guarantee names are unique, or what have you) and concatenate it with the user's identity, then hit that string with a good hash function. MD5 works fine in practice: it doesn't have to be cryptographically secure, it just has to provide good entropy.
We now have a big number which is entropic but which will always be the same for a given user/test pair. Splendid: take the modulus of the number of choices, and give that user the Nth option.
Thus, as long as Bob logs in prior to seeing the A/B test, Bob will always see the same alternative for that test, until you cancel the test.
Now, if users do not log in, it gets a wee bit trickier. You can cookie them the first time you see them, store identifiers in a session, use their IP address (or hash of it) as an identifier, or use other tricks, but it is fundamentally difficult to follow humans across computers without them taking affirmative steps to declare their identity to you.
If this explanation leaves you scratching your head, and you'd prefer seeing it in commented code, I suggest looking at the source code and usage docs of A/Bingo. ( http://abingo.org http://abingo.org ) Disclaimer: I wrote it.
- btilly 17y agoThe hash trick is clever. But what happens when you start with 4 versions, then find that one version is worse than the others? Do you have to start the test over? Is there some kind of rebucketing algorithm you use?
- patio11 17y agoI'd just start the test over. If you want to do rebucketing, see the following Ruby-esque psuedocode: http://pastie.org/610630 http://pastie.org/610630