3 ms·
I've been doing some simulations with bandit algorithms. Early results make them look pretty good (http://www.eanalytica.com/blog/split-test-methodologies-previ
by rfergie 13y ago
I've been doing some simulations with bandit algorithms. Early results make them look pretty good (http://www.eanalytica.com/blog/split-test-methodologies-preview/ http://www.eanalytica.com/blog/split-test-methodologies-prev...).
However, this result could also be explained by the bandit having a lot more variations to play with. I have a lot more work to do on this before I feel sure of an answer
- noelwelsh 13y agoInteresting. I'd like to know more about how you setup your experiments.
- rfergie 13y agoHave chucked the simulation code up on github: https://github.com/richardfergie/split-test-simulator https://github.com/richardfergie/split-test-simulator Current things that I think are wrong with it: * Successive tests should not be independent; picking the wrong option should have repercussions beyond the next test * The distribution of conversion rates from which samples are drawn was picked because it is convenient not because it is correct * Takes no account of resource issues around creating new variations e.g. it is unlikely that people can create a new variation every day but some strategies assume they can. Please let me know your thoughts