4 ms·

Having a cutoff of statistical significance is a side effect of wanting to divide the world into a black-and-white binary categorization. This is a heuristic w

by csense 6y ago

Having a cutoff of statistical significance is a side effect of wanting to divide the world into a black-and-white binary categorization. This is a heuristic which helps things stay within our limited reasoning capacity, cuts down on the noise level from all the activity in the scientific community to focus on what's important, and helps with inherently binary organizational decisions (e.g. should this experiment be published in a journal or not? Should we give this medicine to patients or not?) A better approximation of optimal Bayesian reasoning would be continuous updating of your prior with no cutoff. Even a study with a very weak p-value in the right direction should in theory slightly tweak your priors in favor of the hypothesis. (But you also have to tweak slightly against the hypothesis for studies with very weak p-values in the wrong direction, and compensate for selection effects, e.g. if experiments with underwhelming or wrong-direction p-values are underpublished.) Even weak p-values below the binary decision threshold contain information. It's just that effectively denoising and acting on that information is more difficult and fraught with pitfalls than binary decisions based on strict thresholds. So science mostly standardizes on the safer approach, although there are some people who try to wring precious drops of information out of experiments that fall below the usual statistical cutoffs (often by so-called meta-analysis that combines results of multiple studies).

klodolph 6y ago
And this ties into the whole field of experiment design, and the design of statistical tests. You often conduct an experiment with the goal of making some kind of binary decision. So you figure out what an acceptable risk of type I and type II error would look like, and then you can figure out how much data you need to collect, and where to put the cutoff when analyzing data. But the binary decision “let’s use drug X to treat condition Y”, and the binary decision “let’s investigate the efficacy of drug X on condition Y” each come with radically different tolerances for type I and type II errors.

by csense 6y ago

klodolph 6y ago

And this ties into the whole field of experiment design, and the design of statistical tests. You often conduct an experiment with the goal of making some kind of binary decision. So you figure out what an acceptable risk of type I and type II error would look like, and then you can figure out how much data you need to collect, and where to put the cutoff when analyzing data. But the binary decision “let’s use drug X to treat condition Y”, and the binary decision “let’s investigate the efficacy of drug X on condition Y” each come with radically different tolerances for type I and type II errors.