4 ms·

I guess that's as good a place as any to ask some more general (and naive) question : how come Bayesian methods still haven't taken over the testing market give

by punee 12y ago

I guess that's as good a place as any to ask some more general (and naive) question : how come Bayesian methods still haven't taken over the testing market given their (as perceived by my narrow understanding) advantages?

jcromartie 12y ago
Because Bayesian probability is not intuitive as we (as a society) currently approach statistics. If we start teaching Bayesian probability in grade school then maybe the next generation will "get it" sufficiently to make mainstream applications possible.
- punee 12y ago
  Assuming I'm a marketer who's using A/B testing tools and who's only interested in getting "statistically significant" results, and assuming Bayesian methods provide some advantage in terms of regret minimization that translates to real dollars earned, I feel that I could totally outsource my understanding of the theoretical underpinnings that get me that result. After all, what percentage of Optimizely or VWO users perfectly understands the statistical framework the tools are based on? So I'm not really convinced by that line of reasoning.
  - tel 12y ago
    Bayesian methods don't perform so much better in the situation you just named. They're also more expensive. The really shine in that they provide a really uniform vocabulary of producing new, more sophisticated models while "frequentist" methods usually rely on ingenuity to get to better models. So if you're actively exploring a model space and want to attach a bunch of assumptions and degrees-of-freedom to correspond with a theory you're testing... then Bayesianism is the way to go. Theoretically, marketers are doing that exact process. In practice, they don't see that is a statistical process, though. Maybe someday a tool will bridge that gap successfully... but again, you're unlikely to get a huge advantage with Bayesian methods without that increased work investment.
- theop 12y ago
  I think is more because who is running A/B testing are not knowledgeable in stats, there are many other pitfalls that people fall into (in my personable experience) much worst than not using Bayesian rule, like sampling
  - lynnlinlynn 12y ago
    Yea, I really agree with that statement. There seems to be much bigger problems that most people (starting out or moderately advanced) have with A/B testing. I hate when I see articles that talk about getting amazing results in "just 2 days."
- peatmoss 12y ago
  I'd argue that Bayesian probability is more intuitive to the non-statistically trained. Once people know frequentist statistics, it's more difficult to talk to them about Bayesian results. I think you're right that the future could be more Bayesian if tomorrow's statistical wizards weren't inculcated into a frequentist mindset.
tedsanders 12y ago
Here's my take: 1. People want their formula to tell them things that are impossible to know objectively (like the probability that headline A is better than headline B). 2. They want methods that are easy to use (just plug in some data and press play). 3. And they care more about getting the job done than doing the job right (going from no number to a decent number is more important than going from a decent number to the best number). I think the reason that frequentist statistics are popular is that they meet these three (misguided) desires. Frequentist statistics is full of numbers (like p values) that can easily be confused for the probabilities that people desire. People don't actually care that a p value of 0.05 does NOT mean that the null hypothesis has a 5% chance of being true. They just care that there's an easy objective number that justifies taking an action (like choosing one headline over another). (Of course, this is all my personal speculation.)
Symmetry 12y ago
You can read The Theory That Would Not Die for the gory details but the short of it is that various people objected to the notion that probabilities could be used to encode a degree of certainty; and that the existence of priors made the whole thing unscientific.

by punee 12y ago

jcromartie 12y ago

Because Bayesian probability is not intuitive as we (as a society) currently approach statistics. If we start teaching Bayesian probability in grade school then maybe the next generation will "get it" sufficiently to make mainstream applications possible.

punee 12y ago
Assuming I'm a marketer who's using A/B testing tools and who's only interested in getting "statistically significant" results, and assuming Bayesian methods provide some advantage in terms of regret minimization that translates to real dollars earned, I feel that I could totally outsource my understanding of the theoretical underpinnings that get me that result. After all, what percentage of Optimizely or VWO users perfectly understands the statistical framework the tools are based on? So I'm not really convinced by that line of reasoning.
- tel 12y ago
  Bayesian methods don't perform so much better in the situation you just named. They're also more expensive. The really shine in that they provide a really uniform vocabulary of producing new, more sophisticated models while "frequentist" methods usually rely on ingenuity to get to better models. So if you're actively exploring a model space and want to attach a bunch of assumptions and degrees-of-freedom to correspond with a theory you're testing... then Bayesianism is the way to go. Theoretically, marketers are doing that exact process. In practice, they don't see that is a statistical process, though. Maybe someday a tool will bridge that gap successfully... but again, you're unlikely to get a huge advantage with Bayesian methods without that increased work investment.
theop 12y ago
I think is more because who is running A/B testing are not knowledgeable in stats, there are many other pitfalls that people fall into (in my personable experience) much worst than not using Bayesian rule, like sampling
- lynnlinlynn 12y ago
  Yea, I really agree with that statement. There seems to be much bigger problems that most people (starting out or moderately advanced) have with A/B testing. I hate when I see articles that talk about getting amazing results in "just 2 days."
peatmoss 12y ago
I'd argue that Bayesian probability is more intuitive to the non-statistically trained. Once people know frequentist statistics, it's more difficult to talk to them about Bayesian results. I think you're right that the future could be more Bayesian if tomorrow's statistical wizards weren't inculcated into a frequentist mindset.

tedsanders 12y ago

Here's my take: 1. People want their formula to tell them things that are impossible to know objectively (like the probability that headline A is better than headline B). 2. They want methods that are easy to use (just plug in some data and press play). 3. And they care more about getting the job done than doing the job right (going from no number to a decent number is more important than going from a decent number to the best number). I think the reason that frequentist statistics are popular is that they meet these three (misguided) desires. Frequentist statistics is full of numbers (like p values) that can easily be confused for the probabilities that people desire. People don't actually care that a p value of 0.05 does NOT mean that the null hypothesis has a 5% chance of being true. They just care that there's an easy objective number that justifies taking an action (like choosing one headline over another). (Of course, this is all my personal speculation.)

Symmetry 12y ago

You can read The Theory That Would Not Die for the gory details but the short of it is that various people objected to the notion that probabilities could be used to encode a degree of certainty; and that the existence of priors made the whole thing unscientific.