5 ms·

Maybe it's because my formal math training is not in probability and statistics, but it's so bizarre to me that in a technical situation people would let a phil

by dharmon 10y ago

Maybe it's because my formal math training is not in probability and statistics, but it's so bizarre to me that in a technical situation people would let a philosophical position dictate their approach rather than best tools for the job. Sometimes I'll solve a math problem analytically, and sometimes its easier to do it numerically. But it would be foolish for me to take a hardline stance on one vs the other. Rather I am a more well-rounded, and thus more capable technician because I know the benefits and weaknesses of each approach. If you have small-to-medium sample sizes, then clearly Fisher style statistics will not work very well. On the other hand, even if you have a small sample size, if you don't have some general knowledge to guide your priors, you may very well end up with garbage in a Bayesian approach.

murbard2 10y ago
How do you decide which tool is the best for the job?
- kobeya 10y ago
  The one that yields the simpler solution.
  - dragandj 10y ago
    How do you know which one leads to any good solution, let alone a simple solution?
  - kgwgk 10y ago
    Another option is to choose the one that yields the (more) correct solution.
  - murbard2 10y ago
    How do you know the solution is any good? I'm not going to go all Socrates on you so I'll jump to my point: when discussing bayesian statistics, we're touching something very profound about epistemology that can't be swept under the rug in the name of pragmatism. We're dealing with the core philosophical underpinnings of what it means to "know" something.
loup-vaillant 10y ago
This is different. In math, it doesn't matter which method you're using: all correct methods that yield an answer will yield the same answer. So, the best method is merely the easiest to apply, or the one that'll yield the answer fastest. Bayesian methods are similar: given the entirety of the information you have at your disposal (prior + data), there is one and only one posterior probability distribution over all possible conclusions. No matter how you approach the problem, applying probability theory correctly will yield the same answer. The only drawback is that doing it correctly is often computationally intractable. So we take shortcuts, such as Monte Carlo, hence approximating the correct method, possibly yielding different results depending on the exact nature of the approximation. Frequentist methods however can yield different conclusions depending on who you approach the problem, before you even approximate anything. In part because it sometimes takes into account irrelevant information, such as the researcher's state of mind at the time of the experiment (did I stop at 100 trials because I just got statistical significance, or did I stop at 100 trials because that was decided in advance?). To me, such insanity is hard to fathom. Are frequentist methods that stupid? The glimpse I have got so far indicate they might be.
- yummyfajitas 10y ago
  Frequentist methods are not dependent on the researcher's state of mind, but on what actions he would have taken if things turned out differently. I've got slides that illustrate exactly your example. Look at the graph below: https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2015/slides.html#18 https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2... Your different conditions simply represent the probability of the blue line crossing two different other lines. Frequentism isn't stupid, but you do need to be really smart to use them correctly. At VWO we switched to Bayesian because our customers are marketers, not statisticians.
  - loup-vaillant 10y ago
    > Frequentist methods are not dependent on the researcher's state of mind, but on what actions he would have taken if things turned out differently. That's exactly what I had in mind. It amounts to the same, really. Granted, the researcher's decision process is bloody important: it determines what the researcher will be doing. But once you know what has been done, that decision process has no further influence over the experimental results —only the actual experiment does. Of course, this is all knowing what priors we had in the first place. Since one's priors tend to influence one's decision process, one may chose to ignore one's own priors, and deduce them back from one's decision process. In that narrow sense, different decision processes do yield different conclusions. Going this route amounts to ignore relevant information however, and that would be stupid. > Frequentism isn't stupid, but you do need to be really smart to use them correctly. I believe the "you have to be smart" argument have also been used to attack Bayesian methods on practical grounds. It's a valid attack either way, but a bit weak for my taste. When you apply probability theory, you have to make a logical error to draw the wrong conclusion. The only problem left is prior beliefs, and I don't believe we can escape the need for them. I'm not sure mucking Frequentist methods up requires a logical error. If it indeed doesn't, then we can safely declare Frequentism "unsound" and move on, don't you think? > [VWO slides] Hmm, so Bayesian methods are easier to explain… interesting, thanks for the link.

by dharmon 10y ago

murbard2 10y ago

How do you decide which tool is the best for the job?

kobeya 10y ago
The one that yields the simpler solution.
- dragandj 10y ago
  How do you know which one leads to any good solution, let alone a simple solution?
- kgwgk 10y ago
  Another option is to choose the one that yields the (more) correct solution.
- murbard2 10y ago
  How do you know the solution is any good? I'm not going to go all Socrates on you so I'll jump to my point: when discussing bayesian statistics, we're touching something very profound about epistemology that can't be swept under the rug in the name of pragmatism. We're dealing with the core philosophical underpinnings of what it means to "know" something.

loup-vaillant 10y ago

This is different. In math, it doesn't matter which method you're using: all correct methods that yield an answer will yield the same answer. So, the best method is merely the easiest to apply, or the one that'll yield the answer fastest. Bayesian methods are similar: given the entirety of the information you have at your disposal (prior + data), there is one and only one posterior probability distribution over all possible conclusions. No matter how you approach the problem, applying probability theory correctly will yield the same answer. The only drawback is that doing it correctly is often computationally intractable. So we take shortcuts, such as Monte Carlo, hence approximating the correct method, possibly yielding different results depending on the exact nature of the approximation. Frequentist methods however can yield different conclusions depending on who you approach the problem, before you even approximate anything. In part because it sometimes takes into account irrelevant information, such as the researcher's state of mind at the time of the experiment (did I stop at 100 trials because I just got statistical significance, or did I stop at 100 trials because that was decided in advance?). To me, such insanity is hard to fathom. Are frequentist methods that stupid? The glimpse I have got so far indicate they might be.

yummyfajitas 10y ago
Frequentist methods are not dependent on the researcher's state of mind, but on what actions he would have taken if things turned out differently. I've got slides that illustrate exactly your example. Look at the graph below: https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2015/slides.html#18 https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2... Your different conditions simply represent the probability of the blue line crossing two different other lines. Frequentism isn't stupid, but you do need to be really smart to use them correctly. At VWO we switched to Bayesian because our customers are marketers, not statisticians.
- loup-vaillant 10y ago
  > Frequentist methods are not dependent on the researcher's state of mind, but on what actions he would have taken if things turned out differently. That's exactly what I had in mind. It amounts to the same, really. Granted, the researcher's decision process is bloody important: it determines what the researcher will be doing. But once you know what has been done, that decision process has no further influence over the experimental results —only the actual experiment does. Of course, this is all knowing what priors we had in the first place. Since one's priors tend to influence one's decision process, one may chose to ignore one's own priors, and deduce them back from one's decision process. In that narrow sense, different decision processes do yield different conclusions. Going this route amounts to ignore relevant information however, and that would be stupid. > Frequentism isn't stupid, but you do need to be really smart to use them correctly. I believe the "you have to be smart" argument have also been used to attack Bayesian methods on practical grounds. It's a valid attack either way, but a bit weak for my taste. When you apply probability theory, you have to make a logical error to draw the wrong conclusion. The only problem left is prior beliefs, and I don't believe we can escape the need for them. I'm not sure mucking Frequentist methods up requires a logical error. If it indeed doesn't, then we can safely declare Frequentism "unsound" and move on, don't you think? > [VWO slides] Hmm, so Bayesian methods are easier to explain… interesting, thanks for the link.