4 ms·
I love your posts, and completely agree that Bayesian Bandits are not replacements for A/B tests. To be fair though, the realworld issues like nonstationarity
by emcq 11y ago
I love your posts, and completely agree that Bayesian Bandits are not replacements for A/B tests.
To be fair though, the realworld issues like nonstationarity and delayed feedback are also concerns for A/B tests (which you also bring up in your great post), and you can tweak the bayesian bandits to handle these cases decently.
How does counting sessions instead of visitors avoid delayed feedback? I read your post [0] but dont remember anything about that. Is it just that they say that after a session is completed (which is somewhat nebulous to measure in many cases), then you have all the data you need from the visit?
[0] https://www.chrisstucchio.com/blog/2015/no_free_samples.html https://www.chrisstucchio.com/blog/2015/no_free_samples.html
- yummyfajitas 11y agoAbsolutely agree you can tweak Thompson sampling to handle nonstationarity, periodicity and delayed feedback. I even published the math for one variation of nonstationarity: https://www.chrisstucchio.com/blog/2013/time_varying_conversion_rates.html https://www.chrisstucchio.com/blog/2013/time_varying_convers... (I've also dealt with delayed reactions, but I've never published it, and probably won't publish until I launch it.) Dynamic yield has the delayed feedback problem because users might see a variation in session 1 but convert in session 2 (days later). They "solve" this by doing session level tracking instead of visitor level tracking - the delayed feedback is now only 20 minutes (same session) instead of days. The problem is that session A and session B are now correlated since they are the same visitor. IID is now broken.
- emcq 11y agoMakes sense, thanks for the explanation!