3 ms·
A bit of professional curiosity: I note the site appears to be written in Rails. What did you use for the A/B testing? [Edit: I see below that the answer is A
by patio11 16y ago
A bit of professional curiosity: I note the site appears to be written in Rails. What did you use for the A/B testing? [Edit: I see below that the answer is A/Bingo. I approve, for the obvious reason.] Any opinions on it?
You probably already know this, but the statistical magic behind A/B testing lets you know if A was better than B but not by how much. You can calculate "Hmm, 60% more conversions" but that conclusion is written on water, as I discover with disturbing heaven-smites-you-for-arrogance regularity every time I mention the results of the test that way.
If you sustain the 70% higher level, though, hats off and keep spreading the testing gospel!
- dmix 16y agoI use A/Bingo [1] on CareLogger. Although at work (Learnhub.com) we use Vanity [2]. I found that Vanity splits the participants more equally. I noticed that with A/Bingo one would have 50 more trials than the other ones. Not a big deal unless you check the dashboard constantly. They both do a simple task well so either will work fine. Regarding the sustainability, this is something I've noticed as well. The conversion rates fluctuates heavily depending on the days of the week (middle of the week is best). It swings back and forth but 25% is the new middle ground and not just the good days as it was before. [1] http://www.bingocardcreator.com/abingo/ http://www.bingocardcreator.com/abingo/ [2] http://vanity.labnotes.org/ab_testing.html http://vanity.labnotes.org/ab_testing.html
- patio11 16y agoBoth A/Bingo and Vanity split participants in essentially the same fashion: each new participant is assigned totally randomly. (Not only is it the same effect, the algorithm we use for it is practically identical, too.) This tends to produce a phenomenon well-known to coin flippers: the more coins you flip, the closer the percentage of heads and tails will converge to 50/50 and the farther your counts of heads and tails will diverge from each other.
- dmix 16y agoI figured as much. It's likely that I noticed it because in the environment I use Vanity theres about much more traffic than on carelogger so the numbers even out much more quickly.