3 ms·
Careful. It wasn't always higher for MAB even though the tables shown there make it appear so at first. Those tables are showing the conversion rate during the
by tzs 11y ago
Careful. It wasn't always higher for MAB even though the tables shown there make it appear so at first.
Those tables are showing the conversion rate during the test, up to the time when statistical significance is achieved. You generally then stop the test and go with the winning option for all your traffic.
In the two-way test where the two paths have real conversion rates of 10% and 20%, all of the MAB variations did win. Here is how many conversions there would be after 10000 visitors for that test, and how they compare to the A/B test:
RAND 1988
MAB-10 1997 +9
MAB-24 2001 +13
MAB-50 1996 +8
MAB-90 1994 +6
For the three-way test where the three paths have real rates of 10%, 15%, and 20%, here is how many conversions there would be after 10000 visitors:
RAND 1987
MAB-10 1969 -18
MAB-50 1987 +0
MAB-77 1988 +1
Note that MAB-10 loses compared to RAND this time.
(The third column in the above two tables remains the same if you change 10000 to something else, as long as that something else. MAB-10 beats RAND in the first test by 9 conversions, and loses by 18 conversions in the second test).
- raverbashing 11y agoInteresting This suggest to me that, similarly to a lot of algorithms you might want to change your parameters during training So start with MAB-100 (RAND) and then decrease that % over time
- hythloday 11y ago> up to the time when statistical significance is achieved. You generally then stop the test Just a note, don't literally do this: http://conversionxl.com/statistical-significance-does-not-equal-validity/ http://conversionxl.com/statistical-significance-does-not-eq...
- disgruntledphd2 11y agoJust to reiterate, this violates the assumptions under which you get your p-values. I want an A/B testing tool that won't let you see results until it's done.