5 ms·

Raising threshold will not just reduce probability of false positive result, but also will raise probability of false negative. Social sciences are dealing with

by ordu 8y ago

Raising threshold will not just reduce probability of false positive result, but also will raise probability of false negative. Social sciences are dealing with a complex phenomena and it maybe that there are no simple hypothesis like A -> B, that describes reality with p<0.05. While in reality A causes B, just there are C, D, ..., Z, and some of them also causes B, others works other way and cancel some of others. And some of them works only when Moon is in the right phase. p<0.01 is good when we have a good model of reality which generally works. When we have no good model, there are no good value for p. The trouble is all the hypotheses are lies. The are false. We need more data to find good hypotheses. And we think like "there are useful data, and there are useless, we need to collect useful, while rejecting useless". But we do not know what data is useful, while we have no theory. There is an example from physics I like. Static electricity. Researchers described in their works what causes static electricity. There was a lot of empirical data. But all that data was useless, because the most important part of it didn't get recorded. The most important part was a temporality of phenomena. Static electricity worked some time after charging and then discharged. Why? Because of all the materials are not a perfect insulators, there was a process of electrical discharge, there was voltage and current. It was a link to all other known electical phenomena. But physicists missed it because they had no theory, they didn't knew what is important and what is not. They chased what was shiny, like sparks from static electricity, not the lack of sparks after some time. We are modern people. We are clever. We are using statistics to decide what is important and what is not. Maybe it is a key, but we need to remember that it is not a perfect key.

pps43 8y ago
> The trouble is all the hypotheses are lies The problem with p values is the opposite: in a complex system, null hypothesis is almost never true. Everything affects everything else. It's the magnitude of the effect that is important, because it can be too small to have practical implications. If you do enough studies eventually you'll find a result with p value below 0.05 that will overestimate the magnitude of the effect by a lot, and publish it.
amluto 8y ago
> Raising threshold will not just reduce probability of false positive result, but also will raise probability of false negative. Not really. You raise the probability of an inconclusive result when you could otherwise have gotten a positive. If you interpret p > threshold as “null hypothesis is true”, then you are doing the statistics wrong. In most cases, I think a better model would be to extract an effect size such that an effect larger than the size is ruled out by the study to some degree of confidence. Currently, I read about studies that conclude that “such-and-such had no significant effect detected by this study.” Concretely, this looks like “vaccines had no significant effect on autism risk.” This may be accurate, but it’s lousy. How about “vaccines caused no more than an 0.01% increase in autism, and a bigger study could have set an even tighter limit.” Physicists regularly do this. For example, we know that the universe has no overall curvature to a very good approximation.
- Fomite 8y ago
  We already see this happening in the clinical literature though. Effect estimates that show a positive clinical outcome, but have confidence intervals that barely brush the null are described as having "no impact".
deleted 8y ago
[deleted]
pryce 8y ago
> Raising threshold will not just reduce probability of false positive result, but also will raise probability of false negative. And this article emphasises that huge numbers of scientists and their audiences cannot correctly interpret negatives (false or otherwise). The negative means: "Our data does not appear to show any correlation that met the criteria we chose to determine statistical significance" This negative is frequently misinterpreted as: “We have shown with acceptable confidence that there is no correlation". (which does not inherently follow). Therefore, adjusting the common p value threshold to P<0.01 while not correcting this widely-held cognitive error could (potentially) even worsen the problem, because people will encounter proportionally even more of the 'negative' results that they are already well-established as poor interpreters of.
cameldrv 8y ago
I think you're using a strange definition of p<x here. p<0.01 does not mean that the phenomena is perfectly understood, merely the likelihood that some independent variable affects some dependent variable in some direction. For example, global weather is a hugely complex system which is far from fully understood. Let's take an incredibly simple predictive model of temperature: Still, even with the most simplistic model, say, predict the current temperature by just predicting the average temperature for the current season. That's not going to be very accurate, but if you run it for a year it will almost certainly be p<.01.
nradov 8y ago
In social sciences if the effects aren't clear and readily apparent without quibbling over whether p<0.05 or p<0.01 is the right standard then perhaps the whole thing is a waste of time. If our experimental techniques are insufficient for dealing with multiple factors and complex webs of causality then why bother?
- light_hue_1 8y ago
  How clear or important the effect is has nothing at all to do with a p-value. I can have p < 10^(-10) and still have an effect that is so weak that it's meaningless. The confusion you're having so pervasive and a big part of the problem. You have to use effect sizes in order to measure this.
  - nradov 8y ago
    No I'm not confused by the distinction between p-values and effect sizes, which is why I intentionally used the phrase "clear and readily apparent".
- ordu 8y ago
  Science is a coordinated effort of many people. Scientists needs ability to communicate any partial knowledge they found.

by ordu 8y ago

pps43 8y ago

> The trouble is all the hypotheses are lies The problem with p values is the opposite: in a complex system, null hypothesis is almost never true. Everything affects everything else. It's the magnitude of the effect that is important, because it can be too small to have practical implications. If you do enough studies eventually you'll find a result with p value below 0.05 that will overestimate the magnitude of the effect by a lot, and publish it.

amluto 8y ago

> Raising threshold will not just reduce probability of false positive result, but also will raise probability of false negative. Not really. You raise the probability of an inconclusive result when you could otherwise have gotten a positive. If you interpret p > threshold as “null hypothesis is true”, then you are doing the statistics wrong. In most cases, I think a better model would be to extract an effect size such that an effect larger than the size is ruled out by the study to some degree of confidence. Currently, I read about studies that conclude that “such-and-such had no significant effect detected by this study.” Concretely, this looks like “vaccines had no significant effect on autism risk.” This may be accurate, but it’s lousy. How about “vaccines caused no more than an 0.01% increase in autism, and a bigger study could have set an even tighter limit.” Physicists regularly do this. For example, we know that the universe has no overall curvature to a very good approximation.

Fomite 8y ago
We already see this happening in the clinical literature though. Effect estimates that show a positive clinical outcome, but have confidence intervals that barely brush the null are described as having "no impact".

deleted 8y ago

[deleted]

pryce 8y ago

> Raising threshold will not just reduce probability of false positive result, but also will raise probability of false negative. And this article emphasises that huge numbers of scientists and their audiences cannot correctly interpret negatives (false or otherwise). The negative means: "Our data does not appear to show any correlation that met the criteria we chose to determine statistical significance" This negative is frequently misinterpreted as: “We have shown with acceptable confidence that there is no correlation". (which does not inherently follow). Therefore, adjusting the common p value threshold to P<0.01 while not correcting this widely-held cognitive error could (potentially) even worsen the problem, because people will encounter proportionally even more of the 'negative' results that they are already well-established as poor interpreters of.

cameldrv 8y ago

I think you're using a strange definition of p<x here. p<0.01 does not mean that the phenomena is perfectly understood, merely the likelihood that some independent variable affects some dependent variable in some direction. For example, global weather is a hugely complex system which is far from fully understood. Let's take an incredibly simple predictive model of temperature: Still, even with the most simplistic model, say, predict the current temperature by just predicting the average temperature for the current season. That's not going to be very accurate, but if you run it for a year it will almost certainly be p<.01.

nradov 8y ago

In social sciences if the effects aren't clear and readily apparent without quibbling over whether p<0.05 or p<0.01 is the right standard then perhaps the whole thing is a waste of time. If our experimental techniques are insufficient for dealing with multiple factors and complex webs of causality then why bother?

light_hue_1 8y ago
How clear or important the effect is has nothing at all to do with a p-value. I can have p < 10^(-10) and still have an effect that is so weak that it's meaningless. The confusion you're having so pervasive and a big part of the problem. You have to use effect sizes in order to measure this.
- nradov 8y ago
  No I'm not confused by the distinction between p-values and effect sizes, which is why I intentionally used the phrase "clear and readily apparent".
ordu 8y ago
Science is a coordinated effort of many people. Scientists needs ability to communicate any partial knowledge they found.