6 ms·
If they held the same meeting 20 times, would they reach the same conclusion in 19 of those meetings? On a more serious note, I think that the use of the word
by amluto 11y ago
If they held the same meeting 20 times, would they reach the same conclusion in 19 of those meetings?
On a more serious note, I think that the use of the word "significant" to mean "the effect is reasonably likely to exist by some standard" should be abolished.
Webster's 1913 dictionary says:
> Deserving to be considered; important; momentous; as, a significant event.
Statisticians don't use "significant" to mean important at all -- they use it to mean "I could detect it". This is bad when someone publishes a paper saying "I found that some drug significantly reduces such-and-such" -- this could just mean that they did a HUGE study and found that the drug reliably had some completely unimportant effect. It's much worse when it's negated, though. Think about all the headlines that say that some treatment "did not have a significant effect". This is basically meaningless. I could do a study finding that exercise has no significant effect on fitness, for example, by making the study small enough.
A good friend of mine suggested that statisticians replace "significant" with "discernible". So next time someone does a small study, they might find that "eating fat had no discernible effect on weight gain", and perhaps readers would then ask the obvious question, which is "how hard did you look?".
This would also help people doing very good research make less wishy-washy conclusions. For example, suppose that "vaccines have no discernible effect on autism rates". This is probably true in a number of studies, but it's the wrong analysis. If researchers who did these studies had to state the conclusions in a silly manner like that, maybe they'd find a more useful analysis to do.
Hint: doing big studies just so you can fail to find an effect is nonsensical. Instead, do big studies so you can put a tight upper bound on the effect. Don't tell me that vaccines don't have a significant (or discernible) effect on autism. Tell me that, with 99.9% confidence, you have ruled out the possibility that vaccines have caused more than ten autism cases in the entire history of vaccines, and that, most likely, they've caused no cases whatsoever (or whatever the right numbers are).
Edit: fixed an insignificant typo.
- evolsb 11y agoThere is a significant (pun) difference between statistically significant and economically significant. The conclusion in the drug paper conflates the two. In finance, we could find plenty of statistically significant results (e.g. small cap stocks outperform large cap stocks on Fridays, with a small p-value if you like), but most results were not economically significant--they were not usable for a trading system because they were too small to overcome real-world costs. In short, they weren't meaningful in a real world sense, even though the result was detectable statistically.
- eanzenberg 11y agoThis is extremely important and I'm not surprised the media is either ignorant, doesn't understand, or 'stretches the truth' to adhere to their worldview. Statistical significance has NOTHING to do with magnitude of difference!
- nonbel 11y ago>"Statistical significance has NOTHING to do with magnitude of difference!" Here is the equation for a t-statistic (used for the common t-test): d=mean(a)-mean(b) s=sqrt(var(a) + var(b))/sqrt(n) t=d/s I see the magnitude of difference (d) right in the numerator. The t-value is then compared to the t-distribution to get the tail probability.
- xtacy 11y agoYou are also normalising the difference by the variance, so the t-statistic has no units.
- nonbel 11y agoThe claim the significance has nothing to do with the magnitude of a difference is just wrong though. It clearly does, this information is just merged with other information about the variance and sample size to get the p-value, which is compared to a threshold to get significance.
- astazangasta 11y agoThere is a difference between effect size and significance. Things can be extremely statistically significant and have very small effect sizes. But the problem is less here and more that people don't care to understand the models they are using to judge statistical significance. A p-value is simply a magic wand to wave over the data and bless it. Statisticians may tend to look at the data a lot more qualitatively - a p-value might tell you something, but much more important is: "How accurately have I managed to model this system?" This is the larger problem lurking behind "p-hacking" and other colossal statistical fuck-ups: people don't understand the mathematical models they are applying, the limitations of the data, and often don't care to, as long as the veneer of having 'done something' can be applied. This, again, is probably a product of people cranking out shitty papers to make sure that they keep publishing, to continue eking out grants; which, again, is probably a product of research science being generally underfunded for the demands placed on it.
- chc 11y agoThat's the second definition in Webster's 1913 edition. The first is: > Fitted or designed to signify or make known something; having a meaning; standing as a sign or token; expressive or suggestive; as, a significant word or sound; a significant look. It seems to me that this is the sense in which statisticians talk about significance. It means that the results actually signify something rather than just being meaningless noise.
- amluto 11y agoInteresting. That definition seems like a bit of a stretch in this context to me. Results of trials aren't "fitted or designed" to signify -- they are or are not significant by the p-value standard, and whether they are or are not is random (which is the whole point). In any event, I suspect that, among most currently living English speakers, the second definition is what comes to mind.
- statsaresimple 11y agoTo solve the mystery of the significance of the p-value: If mankind were to make one unique experiment a p-value would be a useful tool, but if many experiments are realized and people hide the results only showing those results that they want to show for self-promotion, then selecting a part of the real information (purposely or not) is a lie.
- cf 11y agoThis is a fantastic point. Sometimes when someone hands me a study I will ask what is the effect size. In some studies, even if there was a discernible effect, there is no hope for it to be anything but a small effect.
- tokenadult 11y agoDon't tell me that vaccines don't have a significant (or discernible) effect on autism. Tell me that, with 99.9% confidence, you have ruled out the possibility that vaccines have caused more than ten autism cases in the entire history of vaccines, and that, most likely, they've caused no cases whatsoever (or whatever the right numbers are). Whoa. Are you making a statement of what you think it would take to convince a hard-line anti-vaccination activist (the most charitable interpretation I can think of for your statement), or are you making that statement on your own behalf? I am chagrined to see this statement in the top comment in this thread. Reading the fine article submitted for discussion is strongly encouraged here on Hacker News, not just reading the headline, and I think you may have missed the part of the article that says, "Indeed, many of the ASA committee’s members argue in their commentaries that the problem isn’t p-values, just the way they’re used — 'failing to adjust them for cherry picking, multiple testing, post-data subgroups and other biasing selection effects,' as Deborah Mayo, a philosopher of statistics at Virginia Tech, puts it. When p-values are treated as a way to sort results into bins labeled significant or not significant, the vast efforts to collect and analyze data are degraded into mere labels, said Kenneth Rothman, an epidemiologist at Boston University." Discernibility of effects versus lack of discernibility of effects is not the main issue here. The main issue is what effects are even possible, plausible, or likely in the first place. One way in which the law of evidence as it has developed in legal procedure over centuries is stronger than the statistical processes used in scientific publications is that the law of evidence remembers that human beings can have biases. I am surprised to see a participant here on Hacker News holding to the strong prior that we must assume that vaccines have caused some cases of autism. Nope. There has never, ever, ever, been any evidence that suggests a causal mechanism such that vaccines could cause even one case of autism. Moreover, there is abundant evidence that the first persons who promoted such claims were trying to abuse the tort law system in England to shake down vaccine manufacturers for payments for imaginary harms.[1] The kind of thinking brought into the discussion here in the comment quoted above doesn't take into account the difference between "evidence-based medicine" and "science-based medicine,"[2] in which clinical trials and other statistical data are supplemented with information from basic science about what priors are plausible and which are plainly implausible. There is no effect discernible from vaccines in increasing rates of autism because there is, on the basis of multiple lines of evidence, no effect that is the tiniest bit plausible for such a claimed harm, and a very plausible line of evidence showing greed and desire for publicity on the part of the corrupt lawyers and doctors who introduced the claim. [1] http://www.bmj.com/content/342/bmj.c5347 http://www.bmj.com/content/342/bmj.c5347 [2] https://www.sciencebasedmedicine.org/about-science-based-medicine/ https://www.sciencebasedmedicine.org/about-science-based-med...
- ryanmonroe 11y ago>Hint: doing big studies just so you can fail to find an effect is nonsensical. Instead, do big studies so you can put a tight upper bound on the effect. Don't tell me that vaccines don't have a significant (or discernible) effect on autism. Tell me that, with 99.9% confidence, you have ruled out the possibility that vaccines have caused more than ten autism cases in the entire history of vaccines, and that, most likely, they've caused no cases whatsoever (or whatever the right numbers are). Checking the bounds of a 100(1-a)% confidence interval is exactly equivalent to checking for a p-value below a%.
- abecedarius 11y agoWhile I agree that 'significant' has a misleading connotation, 'discernible' is also misleading. 'Statistically significant' just isn't any everyday concept, and trying to phrase it as one will encourage people to make mistakes. It's a complex concept: if we tried this experiment under a certain null hypothesis, then it'd be at least this improbable to see a result at least this extreme. The most I'd be willing to cut it down, after so much confusion in its actual use, is "subjunctively improbable", with the null hypothesis and the threshold left implicit. "Eating fat had no subjunctively improbable effect on weight gain." This sounds technical and fiddly, which I think is a feature: if you don't like it, don't base your reporting on a technical, fiddly concept. "Eating fat had no discernible effect on weight gain" sounds like getting evidence against such an effect, but it's compatible with getting evidence in favor, that's just not as strong as some threshold. That evidence could be useful in a meta-analysis, or for a decision when waiting for more information isn't practical or economical, or if the potential gain from trying the nonsignificant treatment is high and the potential loss low. (I've seen "no significant X" abused this way. Nobody should try X -- it's unscientific!)