3 ms·

Sure, but it makes me question these HN comments like "the author is clueless about statistics" and the like.

dekhn 5y ago
the vast majority of scientists are know are clueless about statistics. Biochemisty isn't a field that is strongly dependent on stats, in a way that (for example) genomics is.
- nkurz 5y ago
  "Clueless" is misleading. It seems fair to assume anyone with a PhD in a hard science is probably in the 99%+ percentile for statistical knowledge. Sure, even at this level the statistics often get applied incorrectly, but I'm not sure how well statistical intuition correlates with the number of courses taken. Deep within the linked piece there is a small section entitled 'On “statistical significance”'. It says this: Now, this result does not happen to be “statistically significant.” People often use arguments like “but it’s not statistically significant” to try to discount results. I even hear people with PhDs or MDs say this, because somewhere in their schooling they were falsely taught that results need to be “statistically significant” to be “real.” But even if a result is not statistically significant, absent better data, it still remains the best available estimate for the effect in question. This is just common sense. “Statistically insignificant” does not mean that there is no effect. More on statistical significance here: https://www.nature.com/articles/d41586-019-00857-9 https://www.nature.com/articles/d41586-019-00857-9. As a non-PhD who has (relatively) thought a lot about statistics, and who has previously linked others to that Nature article, this is the sort of balanced statement that makes me think she is not clueless. How about you?
  - dekhn 5y ago
    I've talked to nearly every PhD scientist I've worked with about multi test corrections. It's one of my favorite things. I didn't really understand it, and totally ignored it for many years until it was pointed out, and after that, I noticed that nearly everything I saw published did multiple tests without corrections (combined with very, very large numbers of tests, and fairly loose p--values). I don't think anybody mentioned multi test corrections until I voluntarily took an intro stats class in grad school. BTW, I strongly disagree the quote you have about statistically insignificant data. It cannot be used to say there is no effect, or anything about the direction or magnitude of the effect. I'm not sure what a statement like """somewhere in their schooling they were falsely taught that results need to be “statistically significant” to be “real.”""" even means, significance is a mathematical/probabilistic formalism that has nothing to do with underlying reality, just with the event system you're working on.
    - nkurz 5y ago
      Thanks for the response. I'm confused by your "BTW", though. Are you saying you strongly disagree with her sentiment that "even if a result is not statistically significant, absent better data, it still remains the best available estimate for the effect in question"? Because I feel like this is a useful insight which often gets lost among statisticians: if you have limited information and are forced to make a bet, you probably should bet that whatever actually happened will happen again. True, you should have low confidence in this bet, but you shouldn't throw away information just because 6 times out of 100 (p>.05) it might have occurred purely by chance.
      - dekhn 5y ago
        yeah, I don't think that's a useful insight. I would say that if a result is not statistically significant, you shouldn't consider it to be the "best available estimate" (which is sort of a different concept anyway) unless you're willing to deal with a very high false positive rate. BTW I set my p threshold at 1e-6 for a single test, 0.05 is just crazy. On the other hand, I've worked with systems where nobody cared about the significance, just that the weights that got pushed to production made more money than the last set of weights.
xboxnolifes 5y ago
If the people I talked to in college for a science were any indicator, I'd still believe plenty of scientists are clueless about statistics.

the vast majority of scientists are know are clueless about statistics. Biochemisty isn't a field that is strongly dependent on stats, in a way that (for example) genomics is.

nkurz 5y ago
"Clueless" is misleading. It seems fair to assume anyone with a PhD in a hard science is probably in the 99%+ percentile for statistical knowledge. Sure, even at this level the statistics often get applied incorrectly, but I'm not sure how well statistical intuition correlates with the number of courses taken. Deep within the linked piece there is a small section entitled 'On “statistical significance”'. It says this: Now, this result does not happen to be “statistically significant.” People often use arguments like “but it’s not statistically significant” to try to discount results. I even hear people with PhDs or MDs say this, because somewhere in their schooling they were falsely taught that results need to be “statistically significant” to be “real.” But even if a result is not statistically significant, absent better data, it still remains the best available estimate for the effect in question. This is just common sense. “Statistically insignificant” does not mean that there is no effect. More on statistical significance here: https://www.nature.com/articles/d41586-019-00857-9 https://www.nature.com/articles/d41586-019-00857-9. As a non-PhD who has (relatively) thought a lot about statistics, and who has previously linked others to that Nature article, this is the sort of balanced statement that makes me think she is not clueless. How about you?
- dekhn 5y ago
  I've talked to nearly every PhD scientist I've worked with about multi test corrections. It's one of my favorite things. I didn't really understand it, and totally ignored it for many years until it was pointed out, and after that, I noticed that nearly everything I saw published did multiple tests without corrections (combined with very, very large numbers of tests, and fairly loose p--values). I don't think anybody mentioned multi test corrections until I voluntarily took an intro stats class in grad school. BTW, I strongly disagree the quote you have about statistically insignificant data. It cannot be used to say there is no effect, or anything about the direction or magnitude of the effect. I'm not sure what a statement like """somewhere in their schooling they were falsely taught that results need to be “statistically significant” to be “real.”""" even means, significance is a mathematical/probabilistic formalism that has nothing to do with underlying reality, just with the event system you're working on.
  - nkurz 5y ago
    Thanks for the response. I'm confused by your "BTW", though. Are you saying you strongly disagree with her sentiment that "even if a result is not statistically significant, absent better data, it still remains the best available estimate for the effect in question"? Because I feel like this is a useful insight which often gets lost among statisticians: if you have limited information and are forced to make a bet, you probably should bet that whatever actually happened will happen again. True, you should have low confidence in this bet, but you shouldn't throw away information just because 6 times out of 100 (p>.05) it might have occurred purely by chance.
    - dekhn 5y ago
      yeah, I don't think that's a useful insight. I would say that if a result is not statistically significant, you shouldn't consider it to be the "best available estimate" (which is sort of a different concept anyway) unless you're willing to deal with a very high false positive rate. BTW I set my p threshold at 1e-6 for a single test, 0.05 is just crazy. On the other hand, I've worked with systems where nobody cared about the significance, just that the weights that got pushed to production made more money than the last set of weights.

xboxnolifes 5y ago

If the people I talked to in college for a science were any indicator, I'd still believe plenty of scientists are clueless about statistics.