4 ms·
I mean, it does provide no evidence for the specific hypothesis. It may provide evidence of other hypotheses. So you're right, but you're also wrong, without
by gfodor 7y ago
I mean, it does provide no evidence for the specific hypothesis.
It may provide evidence of other hypotheses.
So you're right, but you're also wrong, without making yourself more clear.
Poor precision in the use of language around clinical data has been a major problem in all of this, in the press especially, where for the most part they clearly don't give a damn and just have an agenda to spin in most cases. Because of this, scientists publishing ought to try to use language that ensures their work cannot be misinterpreted. The notes in the results section here about non-statistically significant aspects to the data fail that test, since an average reader would likely walk away with a false mental model of what the data is showing.
edit: my comment is wrong, i'm tired and haven't had enough coffee. the criticisms below are valid. i do have a point, but won't jam it in here.
- klodolph 7y ago> I mean, it does provide no evidence for the specific hypothesis. Again, this is incorrect. It provides weak evidence, not no evidence. You talk about precise language here but you are conflating “not statistically significant” with “no evidence”. You can’t just handwave the difference between these two things.
- jefftk 7y ago> it does provide no evidence for the specific hypothesis It does provide some evidence: we are more likely to see this result in a world where the hypothesis is true than a world where it is false.
- csense 7y agoHaving a cutoff of statistical significance is a side effect of wanting to divide the world into a black-and-white binary categorization. This is a heuristic which helps things stay within our limited reasoning capacity, cuts down on the noise level from all the activity in the scientific community to focus on what's important, and helps with inherently binary organizational decisions (e.g. should this experiment be published in a journal or not? Should we give this medicine to patients or not?) A better approximation of optimal Bayesian reasoning would be continuous updating of your prior with no cutoff. Even a study with a very weak p-value in the right direction should in theory slightly tweak your priors in favor of the hypothesis. (But you also have to tweak slightly against the hypothesis for studies with very weak p-values in the wrong direction, and compensate for selection effects, e.g. if experiments with underwhelming or wrong-direction p-values are underpublished.) Even weak p-values below the binary decision threshold contain information. It's just that effectively denoising and acting on that information is more difficult and fraught with pitfalls than binary decisions based on strict thresholds. So science mostly standardizes on the safer approach, although there are some people who try to wring precious drops of information out of experiments that fall below the usual statistical cutoffs (often by so-called meta-analysis that combines results of multiple studies).
- klodolph 7y agoAnd this ties into the whole field of experiment design, and the design of statistical tests. You often conduct an experiment with the goal of making some kind of binary decision. So you figure out what an acceptable risk of type I and type II error would look like, and then you can figure out how much data you need to collect, and where to put the cutoff when analyzing data. But the binary decision “let’s use drug X to treat condition Y”, and the binary decision “let’s investigate the efficacy of drug X on condition Y” each come with radically different tolerances for type I and type II errors.