4 ms·

This defense of P-values is akin to how a certain type of computer programmer tends to defend bad design decisions, like "if you accidentally typed `rm -rf /` a

This defense of P-values is akin to how a certain type of computer programmer tends to defend bad design decisions, like "if you accidentally typed `rm -rf /` and it wiped out your entire system, then you're an idiot, learn to use your tools". Thing is: even statistics professors often don't manage to interpret P-values and confidence intervals the way they should. (See e.g. "Students’ misconceptions of statistical inference: A review of the empirical evidence from research on statistics education") You're almost automatically forced into a sort of double talk where you write about P(D|H0) but deep down you'd really like your readers to think it's about 1-P(HA|D). When a tool can be used correctly (if handled very delicately and by a true expert) but practically encourages you to abuse it, at a certain point "it's not the tool, it's you" stops being a convincing excuse, and you've just got to say: fuck it, we need better tools.

lmm 12y ago
At some point research is going to be hard - after all, if it were easy to discover someone would have done so already. Sure, we should always strive to improve our tools - but that's also hard. Throwing up our hands and stopping research is not an option. So what do we do? Is there a better alternative? If the P-value is the frying pan then Bayesianism is the fire; you don't get a standard library of tools appropriate to different situations, you get one powerful tool that has to be applied correctly, and for the part that really matters in the contentious cases - choice of prior - you're almost completely on your own. Release your raw data. Use simple, standard analyses; do them once and do them right, and stay the hell away from subgroup analysis. It really isn't too hard to get the statistics right - it's just very easy to let yourself slip, especially when your future funding/career depends on it.
- stdbrouw 12y ago
  Power analysis and confidence intervals have their own problems, but they're already a vast improvement over p-values. No need to go all-out Bayesian straight away – though certainly, if we can get the software to a point where that would be possible for non-statisticians, that'd be awesome. (Bayes factors help when you don't want to fix a prior.) For anyone who's interested to read more about this middle ground approach, "The Essential Guide to Effect Sizes" by Paul Ellis is an awesome introduction not just to power analysis but to better statistical practice and statistical reporting that sticks to the frequentist tradition. Also, I don't feel subgroup analysis is necessarily the problem, at least not when you apply any of the available corrections for familywise error rates (Bonferroni and Tukey's HSD being two of the more well known). This often comes up in medical research, e.g. http://www.badscience.net/2009/04/a-frankly-thin-contrivance-for-writing-on-the-fascinating-issue-of-subgroup-analysis/ http://www.badscience.net/2009/04/a-frankly-thin-contrivance... -- but in those cases subgroup analyses are performed with bad intentions, a whole other problem! A bigger issue, perhaps, is drawing inferences from hypotheses that came up when analyzing the data in the first place – it's very appealing but it's also wonderfully circular in its logic and why in machine learning circles it's become almost self-evident to split up the data into training and test sets.
  - IndianAstronaut 12y ago
    >machine learning circles it's become almost self-evident to split up the data into training and test sets. This is great for predictive modeling but how does this work for analysis of experiments?
    - stdbrouw 12y ago
      Not experiments, but observational data is often analyzed using regression, the poor man's predictive modeling :-)

lmm 12y ago

At some point research is going to be hard - after all, if it were easy to discover someone would have done so already. Sure, we should always strive to improve our tools - but that's also hard. Throwing up our hands and stopping research is not an option. So what do we do? Is there a better alternative? If the P-value is the frying pan then Bayesianism is the fire; you don't get a standard library of tools appropriate to different situations, you get one powerful tool that has to be applied correctly, and for the part that really matters in the contentious cases - choice of prior - you're almost completely on your own. Release your raw data. Use simple, standard analyses; do them once and do them right, and stay the hell away from subgroup analysis. It really isn't too hard to get the statistics right - it's just very easy to let yourself slip, especially when your future funding/career depends on it.

stdbrouw 12y ago
Power analysis and confidence intervals have their own problems, but they're already a vast improvement over p-values. No need to go all-out Bayesian straight away – though certainly, if we can get the software to a point where that would be possible for non-statisticians, that'd be awesome. (Bayes factors help when you don't want to fix a prior.) For anyone who's interested to read more about this middle ground approach, "The Essential Guide to Effect Sizes" by Paul Ellis is an awesome introduction not just to power analysis but to better statistical practice and statistical reporting that sticks to the frequentist tradition. Also, I don't feel subgroup analysis is necessarily the problem, at least not when you apply any of the available corrections for familywise error rates (Bonferroni and Tukey's HSD being two of the more well known). This often comes up in medical research, e.g. http://www.badscience.net/2009/04/a-frankly-thin-contrivance-for-writing-on-the-fascinating-issue-of-subgroup-analysis/ http://www.badscience.net/2009/04/a-frankly-thin-contrivance... -- but in those cases subgroup analyses are performed with bad intentions, a whole other problem! A bigger issue, perhaps, is drawing inferences from hypotheses that came up when analyzing the data in the first place – it's very appealing but it's also wonderfully circular in its logic and why in machine learning circles it's become almost self-evident to split up the data into training and test sets.
- IndianAstronaut 12y ago
  >machine learning circles it's become almost self-evident to split up the data into training and test sets. This is great for predictive modeling but how does this work for analysis of experiments?
  - stdbrouw 12y ago
    Not experiments, but observational data is often analyzed using regression, the poor man's predictive modeling :-)