3 ms·

Thanks for the response. I'm confused by your "BTW", though. Are you saying you strongly disagree with her sentiment that "even if a result is not statisticall

by nkurz 5y ago

Thanks for the response. I'm confused by your "BTW", though. Are you saying you strongly disagree with her sentiment that "even if a result is not statistically significant, absent better data, it still remains the best available estimate for the effect in question"? Because I feel like this is a useful insight which often gets lost among statisticians: if you have limited information and are forced to make a bet, you probably should bet that whatever actually happened will happen again. True, you should have low confidence in this bet, but you shouldn't throw away information just because 6 times out of 100 (p>.05) it might have occurred purely by chance.

dekhn 5y ago
yeah, I don't think that's a useful insight. I would say that if a result is not statistically significant, you shouldn't consider it to be the "best available estimate" (which is sort of a different concept anyway) unless you're willing to deal with a very high false positive rate. BTW I set my p threshold at 1e-6 for a single test, 0.05 is just crazy. On the other hand, I've worked with systems where nobody cared about the significance, just that the weights that got pushed to production made more money than the last set of weights.