4 ms·

I've often told people in the passing that "if you have 20 parameters and p=0.05 you should expect with random data to have something register as significant".

I've often told people in the passing that "if you have 20 parameters and p=0.05 you should expect with random data to have something register as significant". Looks like OP has beaten me to it to illustrate this. [1] [1] http://www.jerrydallal.com/LHSP/coffee.htm http://www.jerrydallal.com/LHSP/coffee.htm

Nokinside 7y ago
If you have 20 parameters, you should probably compare the effect sizes first. If there are no outliers, p=0.05 or p=0.001 can't make the result important. https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1583913 https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1... > no p-value can reveal the plausibility, presence, truth, or importance of an association or effect. Therefore, a label of statistical significance does not mean or imply that an association or effect is highly probable, real, true, or important. Nor does a label of statistical nonsignificance lead to the association or effect being improbable, absent, false, or unimportant. Yet the dichotomization into “significant” and “not significant” is taken as an imprimatur of authority on these characteristics. In a world without bright lines, on the other hand, it becomes untenable to assert dramatic differences in interpretation from inconsequential differences in estimates. As Gelman and Stern (2006 Gelman, A., and Stern, H. (2006), “The Difference Between ‘Significant’ and ‘Not Significant’ Is Not Itself Statistically Significant,” The American Statistician, 60, 328–331. DOI: 10.1198/000313006X152649. ) famously observed, the difference between “significant” and “not significant” is not itself statistically significant.
- brookhaven_dude 7y ago
  What do you say about neural networks with thousands of parameters?
  - atlasair 7y ago
    Neural networks are just used for prediction, not inference.

Nokinside 7y ago

If you have 20 parameters, you should probably compare the effect sizes first. If there are no outliers, p=0.05 or p=0.001 can't make the result important. https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1583913 https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1... > no p-value can reveal the plausibility, presence, truth, or importance of an association or effect. Therefore, a label of statistical significance does not mean or imply that an association or effect is highly probable, real, true, or important. Nor does a label of statistical nonsignificance lead to the association or effect being improbable, absent, false, or unimportant. Yet the dichotomization into “significant” and “not significant” is taken as an imprimatur of authority on these characteristics. In a world without bright lines, on the other hand, it becomes untenable to assert dramatic differences in interpretation from inconsequential differences in estimates. As Gelman and Stern (2006 Gelman, A., and Stern, H. (2006), “The Difference Between ‘Significant’ and ‘Not Significant’ Is Not Itself Statistically Significant,” The American Statistician, 60, 328–331. DOI: 10.1198/000313006X152649. ) famously observed, the difference between “significant” and “not significant” is not itself statistically significant.

brookhaven_dude 7y ago
What do you say about neural networks with thousands of parameters?
- atlasair 7y ago
  Neural networks are just used for prediction, not inference.