6 ms·

BTW, asking the question "what is the difference between power and significance/p-value" is a good litmus test for detecting insufficient competence in statisti

BTW, asking the question "what is the difference between power and significance/p-value" is a good litmus test for detecting insufficient competence in statistics. As the author puts it, unless you know this stuff well you shouldn't really be juding the "scientific" numbers. In fact you're probably (probably!) better off building a good intuition within the domain and then improvising the tests, in the sense that this way you're less likely to arrive at bogus conclusions, as opposed to someone who thinks they'll just apply some formula like this was a high school maths problem - here we plug in some magic that speaks back the truth. I've seen it time and again that (especially senior) people can rapidly and accurately judge the results of things based on very "un-scientific" improvised methods and numbers. YMMV.

sokoloff 5y ago
/me quickly googling “what is the difference between power and significance/p-value?” because I know I lack competence in statistics.
- tel 5y ago
  Broadly, there are two kinds of error: false positives and false negatives. A false positive arises because reality conspired to produce what you consider to be an anomalous result. False negatives arise because your detector wasn't sensitive enough to notice the anomalous result. The standard statistical practice is to fix your risk of false positives through the preemptive choice of a p-value threshold and then to attempt to minimize your risk of false negatives through increases in the "power" of your experiment. But, honestly, these two are in tension with one another. There are lots of possible choices. It's important to know the difference, though. It changes how you interpret results. A positive result might be doubted because you, as a reader, would have preferred more protection against false positives, and, thusly, a smaller p-value threshold. A negative result might be doubted because you believe that the experimenter under-powered their experiment. You might also, in a repeating experimental process, wish to obtain greater power by sacrificing false-positive protection. This is as easy as changing the p-value threshold. A last perspective is that in the typical scientific practice, you basically control false-positives through pushing the p-value down and making the experiment more challenging (reducing power). Then, you re-establish power through more expensive experiments (use better tools, run the experiment for longer, improve controls, improve data efficiency). Often the "typical" p-value is set by the community in relation to what level of false-positive the overall community is willing to tolerate in publication. That's why psychologists are happy with a 5% threshold whereas physicists might seek a threshold of 10^-3 or smaller. The physicists are upholding themselves to a much more expensive experimental standard and would vastly prefer false negatives.
clircle 5y ago
Power is a pre-test measure, and a p-value is a post test measure.
leephillips 5y ago
All you have to do is ask an author of a paper in medicine or psychology that quotes a p-value to define p-value for you. That will open your eyes if you’re not already clued in.

sokoloff 5y ago

/me quickly googling “what is the difference between power and significance/p-value?” because I know I lack competence in statistics.

tel 5y ago
Broadly, there are two kinds of error: false positives and false negatives. A false positive arises because reality conspired to produce what you consider to be an anomalous result. False negatives arise because your detector wasn't sensitive enough to notice the anomalous result. The standard statistical practice is to fix your risk of false positives through the preemptive choice of a p-value threshold and then to attempt to minimize your risk of false negatives through increases in the "power" of your experiment. But, honestly, these two are in tension with one another. There are lots of possible choices. It's important to know the difference, though. It changes how you interpret results. A positive result might be doubted because you, as a reader, would have preferred more protection against false positives, and, thusly, a smaller p-value threshold. A negative result might be doubted because you believe that the experimenter under-powered their experiment. You might also, in a repeating experimental process, wish to obtain greater power by sacrificing false-positive protection. This is as easy as changing the p-value threshold. A last perspective is that in the typical scientific practice, you basically control false-positives through pushing the p-value down and making the experiment more challenging (reducing power). Then, you re-establish power through more expensive experiments (use better tools, run the experiment for longer, improve controls, improve data efficiency). Often the "typical" p-value is set by the community in relation to what level of false-positive the overall community is willing to tolerate in publication. That's why psychologists are happy with a 5% threshold whereas physicists might seek a threshold of 10^-3 or smaller. The physicists are upholding themselves to a much more expensive experimental standard and would vastly prefer false negatives.

clircle 5y ago

Power is a pre-test measure, and a p-value is a post test measure.

leephillips 5y ago

All you have to do is ask an author of a paper in medicine or psychology that quotes a p-value to define p-value for you. That will open your eyes if you’re not already clued in.