4 ms·

This is an interesting problem. There are so many ways to perform A/B testing incorrectly that I expect most A/B tests performed in the wild yield questionable

This is an interesting problem. There are so many ways to perform A/B testing incorrectly that I expect most A/B tests performed in the wild yield questionable results. I think there is a lot of pseudo-scientific talk around A/B testing that leads people to either put too much credence in their results, or, paradoxically, avoid advanced techniques due to fear their results will be inaccurate. I'm coming to the conclusion that one should treat A/B testing from more of an economic point-of-view. You have so many views this month -- how are you going to spend them on tests? I think one should look at A/B tests as a source of information, not of truth, and thus avoid chasing after statistical rigor that won't, in fact, be realised in most cases.

rfergie 12y ago
You should look at multi-armed bandits - this approach is pretty much what you have reached in your conclusion and a lot of work has been done on them
- noelwelsh 12y ago
  Funnily enough, my starting point to get into A/B testing was research on multiarmed bandits. Update: I should add I think the issues are more complex than simply using, say, Thompson sampling.
  - rfergie 12y ago
    Ha! I should have read your profile before responding.
btilly 12y ago
I'm coming to the conclusion that one should treat A/B testing from more of an economic point-of-view. You have so many views this month -- how are you going to spend them on tests? I'm curious, did our conversations a few years ago help shape this opinion? My guess is that we've both moved somewhat towards a middle position.
- noelwelsh 12y ago
  Yes, I think so. It might be of interest to you that there is now some published work on bandit algorithms for when all you have is ranking, not absolute ordering. This is close to what we were talking about: http://arxiv.org/pdf/1312.3393.pdf http://arxiv.org/pdf/1312.3393.pdf

rfergie 12y ago

You should look at multi-armed bandits - this approach is pretty much what you have reached in your conclusion and a lot of work has been done on them

noelwelsh 12y ago
Funnily enough, my starting point to get into A/B testing was research on multiarmed bandits. Update: I should add I think the issues are more complex than simply using, say, Thompson sampling.
- rfergie 12y ago
  Ha! I should have read your profile before responding.

btilly 12y ago

I'm coming to the conclusion that one should treat A/B testing from more of an economic point-of-view. You have so many views this month -- how are you going to spend them on tests? I'm curious, did our conversations a few years ago help shape this opinion? My guess is that we've both moved somewhat towards a middle position.

noelwelsh 12y ago
Yes, I think so. It might be of interest to you that there is now some published work on bandit algorithms for when all you have is ranking, not absolute ordering. This is close to what we were talking about: http://arxiv.org/pdf/1312.3393.pdf http://arxiv.org/pdf/1312.3393.pdf