3 ms·
Further questions. 1. My experience is that this assumption can be significantly mistaken. I have seen significant daily and weekly variations in conversion r
by btilly 15y ago
Further questions.
1. My experience is that this assumption can be significantly mistaken. I have seen significant daily and weekly variations in conversion rates. (They average out after a bit, but they fluctuate.) Still a small tweak could help.
2. Please do look into this one, the difference is not small. I first encountered the differential timing issue with a test of the benefits of adding a phone touch point on top of an email cycle. That extra contact moved conversions up by weeks. So after the first month it looked fantastic, but then slowly degraded over time. (There was improvement, but not enough to justify the expense.) A careful cohort analysis showed that it was not worthwhile at a point where a naive A/B test was still showing the extra touchpoint winning by a very significant margin.
3. That's reasonable.
4. In practice during A/B testing people don't worry about those things either. They just start the test, and later declare a winner. (People are much more sloppy about it than they theoretically should be...)
- noelwelsh 15y ago1. Let me be a bit more precise. The basic version of Myna assumes average reward is constant over the entire time it is active. In the language of statistics, so long as the process is stationary we're ok. We have some somewhat experimental additions that try to look for slowly changing average reward and compensate for this. I say somewhat experimental as we haven't been able to really empirically validate performance yet. In the literature there are a few different approaches to handling non-stationary systems, and I imagine this is an area we'll address further as we get Myna into more complex environments. 2. Interesting. I'll investigate when I get time. 4. Indeed. In a sense the maths is irrelevant. If you can get people actually measuring the effect of their changes the battle is mostly won. We're certainly finding that it is the ease of use that is most important to our users (echoing patio11's comment below). On the other hand, once you take the human out of the loop you open the door to much more interesting applications and this is where I think the real benefit of Myna will be.