3 ms·

Here's a few suggestions about better alternatives to Science using statistics: http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequently_subjective/ h

by khafra 17y ago

Here's a few suggestions about better alternatives to Science using statistics: http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequently_subjective/ http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequ...

scott_s 17y ago
The obvious fix is to (a) require raw data to be published; (b) require journals to accept papers before the experiment is performed, with the advance paper including a specification of what statistics were selected in advance to be run on the results; (c) raising the standard "significance" level to p<0.0001; and (d) junking all the damned overcomplicated status-seeking impressive nonsense of classical statistics and going to simple understandable Bayesian likelihoods. Suggestion b is both radical and very thought provoking. Which, at this point, I expect from Yudkowsky.
- babar 17y ago
  I think in some fields achieving (c) is almost impossible - the n number needed would be too expensive for most labs working with animal models, for instance. (b) would be interesting, especially if it led to a culture where "failed" results are still worth publishing because they disprove something. I think (a) is the one worth focusing on first, since that seems the most achievable and would let you back-apply some of the other improvements at a later date.
- lutorm 17y ago
  Interesting. The idea that null results are not worth publishing has since the very beginning struck me as one of the most fundamentally flawed ideas in science. Interestingly, it seems to be very domain-dependent. In my field (astrophysics), null results are fairly frequently published, but I've heard that in other fields it's totally impossible to do.
  - argv_empty 17y ago
    I've run across one or two compiler optimization papers where the conclusion given was that the proposed technique didn't work out so well, but on the whole, it seems it applies there as well. I agree that it's a problem -- if a null result is not published, other people will probably waste years making the same mistakes.
    - lutorm 17y ago
      Yeah, the wasted repeated effort was what I initially thought about, too. The argument about how compilations of results will be systematically skewed by certain results not being published is perhaps even more persuasive, because it doesn't just lead to wasted effort but to incorrect results.

by khafra 17y ago

scott_s 17y ago

The obvious fix is to (a) require raw data to be published; (b) require journals to accept papers before the experiment is performed, with the advance paper including a specification of what statistics were selected in advance to be run on the results; (c) raising the standard "significance" level to p<0.0001; and (d) junking all the damned overcomplicated status-seeking impressive nonsense of classical statistics and going to simple understandable Bayesian likelihoods. Suggestion b is both radical and very thought provoking. Which, at this point, I expect from Yudkowsky.

babar 17y ago
I think in some fields achieving (c) is almost impossible - the n number needed would be too expensive for most labs working with animal models, for instance. (b) would be interesting, especially if it led to a culture where "failed" results are still worth publishing because they disprove something. I think (a) is the one worth focusing on first, since that seems the most achievable and would let you back-apply some of the other improvements at a later date.
lutorm 17y ago
Interesting. The idea that null results are not worth publishing has since the very beginning struck me as one of the most fundamentally flawed ideas in science. Interestingly, it seems to be very domain-dependent. In my field (astrophysics), null results are fairly frequently published, but I've heard that in other fields it's totally impossible to do.
- argv_empty 17y ago
  I've run across one or two compiler optimization papers where the conclusion given was that the proposed technique didn't work out so well, but on the whole, it seems it applies there as well. I agree that it's a problem -- if a null result is not published, other people will probably waste years making the same mistakes.
  - lutorm 17y ago
    Yeah, the wasted repeated effort was what I initially thought about, too. The argument about how compilations of results will be systematically skewed by certain results not being published is perhaps even more persuasive, because it doesn't just lead to wasted effort but to incorrect results.