4 ms·
Do you think they pre registered their study into the effect of pre registered studies on null hypothesis rate?
by salty_biscuits 8y ago
Do you think they pre registered their study into the effect of pre registered studies on null hypothesis rate?
- hn_throwaway_99 8y agoThis was a meta-analysis, not a study of its own.
- jeremyjh 8y agoWhat difference does that make?
- laGrenouille 8y agoThe point, I think, is that it means something slightly different to per-register an hypothesis that depends on retrospective data. When you register to hypothesis, the data already exist and one could (in theory) have already started looking at it. One would hope that researchers would not explicitly cheat like this, but it would be hard for experts in the fields to not have some sense of the trends seen within their field prior to running the meta analysis.
- escherplex 8y agoYeah but of course hypotheses don't arise ex nihilo but from a catalog of focused perceptions catalyzing conceptions which may suggest hypotheses encompassing novel ordering functions, innovations, ameliorations (or whatnot) depending on the field of study. If an individual considers a hypothesis of sufficient potential pragmatic value to research and document then, objectively, if negative results manifest these too need be reported as negative data germane to other researchers in the field. Problem is current academic culture is predisposed to reward only success. If your beautiful theory is murdered by an ugly fact (Huxley) then you don't get your doctorate or your departmental status diminishes since only positive results are reinforced. From this arises the temptation to be disingenuous or outright prevaricate, if you can get away with it. Does seem a cultural change is in order. Registering hypotheses which guarantees publication regardless of results may be useful.
- JumpCrisscross 8y ago> One would hope that researchers would not explicitly cheat like this The broad problem isn’t cheating, it’s people not publishing null results. That applies equally to experiments and meta-analyses.
- aidenn0 8y agoExcept a null result would have been interesting here.
- akvadrako 8y agoWhat the poster above meant is that if you do a meta-analysis on already published results you could already know if it's a null result or not. If you are inclined to not publish null results, you simply won't "preregister". Of course to be a proper study they should pre-register and only report on future publications.
- jeremyjh 8y agoYou are right about what the problem is, but cheating in this context would mean registering a hypothesis after you've already looked at the data and assured yourself you'll get a positive result. That in my mind is even worse than current status-quo.
- danaris 8y agoBoth are problems. Both are addressed by pre-registration.
- hannob 8y agoYou can totally p-hack a meta-analysis, you have many degrees of freedom in interpreting the data. Preregistration of a metaanalysis makes sense and should be standard practice.
- Bartweiss 8y agoThe whole framing of meta-analyses as the pinnacle of reliability is sort of concerning. They're a huge help with messy questions like "what the hell is up with priming?", but between choosing which studies to include, standardizing their controls, normalizing their results, etc. it's amazing how far they can be skewed. One thing I don't know: what does preregistration even look like for a metaanalysis? You can state some conditions up front, but a lot of the degrees of freedom only come into play once you're deep in the work looking at specific studies.
- skybrian 8y agoAccording to the article, they did not, but they plan to do another study with more data and preregister it.