4 ms·

Major problem with "significance" is that in some areas of research (say psychology), it's possible to gather lots of data(let 1000 people fill complex question

by Hitton 7y ago

Major problem with "significance" is that in some areas of research (say psychology), it's possible to gather lots of data(let 1000 people fill complex questionnaire) and from that data to fish for theory that's significant (in your sample Republicans might have been dumber than Democrats). But given the size of the sample and number of theories you test, you are bound to find something significant even if that isn't true. Tightening significance threshold just makes this fishing more difficult, Bayesian reasoning doesn't help much either, because you have to guestimate reasonable priors. What does really help against fishing like this is requirement to preregister your studies.

amichal 7y ago
Not a statistician myself but in a visualization project we did a number of years ago that needed a quick fix for this we used https://en.wikipedia.org/wiki/Bonferroni_correction https://en.wikipedia.org/wiki/Bonferroni_correction.
- buboard 7y ago
  The bonferroni correction is a post-test for testing between multiple pairs in groups, it doesnt really fix the problem of cherrypicking hypotheses
bonoboTP 7y ago
Many people talk about preregistration, but I'm not sure it would result in the hoped benefits. What I predict would happen: - either lots of studies are allowed to preregister, most of which cannot reject the null hypothesis. You end up with a lot of "boring" null-result papers in those high-profile journals that nobody gets excited about and nobody gets promoted for and no media coverage happens, bad marketing for universities and research centers. - or there would be a strict filter for the pre-registration, so that researchers cannot chase their gut intuitions, some authority would need to approve the study even before it's done. This hinders research and hinders the dissemination of truly unexpected discoveries. The human incentives are way deeper than any one solution could touch on. The whole science funding structure requires flashy and sexy results that are just not possible to produce on that scale.
- MaxBarraclough 7y ago
  > or there would be a strict filter for the pre-registration, so that researchers cannot chase their gut intuitions, some authority would need to approve the study even before it's done I see two things going on here. Having additional 'oversight' for the scientific validity of a planned experiment, is presumably a good thing. I don't see that it would be necessary to set out to prevent researchers investigating their intuitions.

by Hitton 7y ago

amichal 7y ago

Not a statistician myself but in a visualization project we did a number of years ago that needed a quick fix for this we used https://en.wikipedia.org/wiki/Bonferroni_correction https://en.wikipedia.org/wiki/Bonferroni_correction.

buboard 7y ago
The bonferroni correction is a post-test for testing between multiple pairs in groups, it doesnt really fix the problem of cherrypicking hypotheses

bonoboTP 7y ago

Many people talk about preregistration, but I'm not sure it would result in the hoped benefits. What I predict would happen: - either lots of studies are allowed to preregister, most of which cannot reject the null hypothesis. You end up with a lot of "boring" null-result papers in those high-profile journals that nobody gets excited about and nobody gets promoted for and no media coverage happens, bad marketing for universities and research centers. - or there would be a strict filter for the pre-registration, so that researchers cannot chase their gut intuitions, some authority would need to approve the study even before it's done. This hinders research and hinders the dissemination of truly unexpected discoveries. The human incentives are way deeper than any one solution could touch on. The whole science funding structure requires flashy and sexy results that are just not possible to produce on that scale.

MaxBarraclough 7y ago
> or there would be a strict filter for the pre-registration, so that researchers cannot chase their gut intuitions, some authority would need to approve the study even before it's done I see two things going on here. Having additional 'oversight' for the scientific validity of a planned experiment, is presumably a good thing. I don't see that it would be necessary to set out to prevent researchers investigating their intuitions.