5 ms·

This is different. In math, it doesn't matter which method you're using: all correct methods that yield an answer will yield the same answer. So, the best meth

This is different. In math, it doesn't matter which method you're using: all correct methods that yield an answer will yield the same answer. So, the best method is merely the easiest to apply, or the one that'll yield the answer fastest. Bayesian methods are similar: given the entirety of the information you have at your disposal (prior + data), there is one and only one posterior probability distribution over all possible conclusions. No matter how you approach the problem, applying probability theory correctly will yield the same answer. The only drawback is that doing it correctly is often computationally intractable. So we take shortcuts, such as Monte Carlo, hence approximating the correct method, possibly yielding different results depending on the exact nature of the approximation. Frequentist methods however can yield different conclusions depending on who you approach the problem, before you even approximate anything. In part because it sometimes takes into account irrelevant information, such as the researcher's state of mind at the time of the experiment (did I stop at 100 trials because I just got statistical significance, or did I stop at 100 trials because that was decided in advance?). To me, such insanity is hard to fathom. Are frequentist methods that stupid? The glimpse I have got so far indicate they might be.

yummyfajitas 10y ago
Frequentist methods are not dependent on the researcher's state of mind, but on what actions he would have taken if things turned out differently. I've got slides that illustrate exactly your example. Look at the graph below: https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2015/slides.html#18 https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2... Your different conditions simply represent the probability of the blue line crossing two different other lines. Frequentism isn't stupid, but you do need to be really smart to use them correctly. At VWO we switched to Bayesian because our customers are marketers, not statisticians.
- loup-vaillant 10y ago
  > Frequentist methods are not dependent on the researcher's state of mind, but on what actions he would have taken if things turned out differently. That's exactly what I had in mind. It amounts to the same, really. Granted, the researcher's decision process is bloody important: it determines what the researcher will be doing. But once you know what has been done, that decision process has no further influence over the experimental results —only the actual experiment does. Of course, this is all knowing what priors we had in the first place. Since one's priors tend to influence one's decision process, one may chose to ignore one's own priors, and deduce them back from one's decision process. In that narrow sense, different decision processes do yield different conclusions. Going this route amounts to ignore relevant information however, and that would be stupid. > Frequentism isn't stupid, but you do need to be really smart to use them correctly. I believe the "you have to be smart" argument have also been used to attack Bayesian methods on practical grounds. It's a valid attack either way, but a bit weak for my taste. When you apply probability theory, you have to make a logical error to draw the wrong conclusion. The only problem left is prior beliefs, and I don't believe we can escape the need for them. I'm not sure mucking Frequentist methods up requires a logical error. If it indeed doesn't, then we can safely declare Frequentism "unsound" and move on, don't you think? > [VWO slides] Hmm, so Bayesian methods are easier to explain… interesting, thanks for the link.

yummyfajitas 10y ago

Frequentist methods are not dependent on the researcher's state of mind, but on what actions he would have taken if things turned out differently. I've got slides that illustrate exactly your example. Look at the graph below: https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2015/slides.html#18 https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2... Your different conditions simply represent the probability of the blue line crossing two different other lines. Frequentism isn't stupid, but you do need to be really smart to use them correctly. At VWO we switched to Bayesian because our customers are marketers, not statisticians.

loup-vaillant 10y ago
> Frequentist methods are not dependent on the researcher's state of mind, but on what actions he would have taken if things turned out differently. That's exactly what I had in mind. It amounts to the same, really. Granted, the researcher's decision process is bloody important: it determines what the researcher will be doing. But once you know what has been done, that decision process has no further influence over the experimental results —only the actual experiment does. Of course, this is all knowing what priors we had in the first place. Since one's priors tend to influence one's decision process, one may chose to ignore one's own priors, and deduce them back from one's decision process. In that narrow sense, different decision processes do yield different conclusions. Going this route amounts to ignore relevant information however, and that would be stupid. > Frequentism isn't stupid, but you do need to be really smart to use them correctly. I believe the "you have to be smart" argument have also been used to attack Bayesian methods on practical grounds. It's a valid attack either way, but a bit weak for my taste. When you apply probability theory, you have to make a logical error to draw the wrong conclusion. The only problem left is prior beliefs, and I don't believe we can escape the need for them. I'm not sure mucking Frequentist methods up requires a logical error. If it indeed doesn't, then we can safely declare Frequentism "unsound" and move on, don't you think? > [VWO slides] Hmm, so Bayesian methods are easier to explain… interesting, thanks for the link.