2 ms·
I believe that's where the original post's >given enough time. comes in. One nice thing about simulation based research is that it is often (more) reproducib
by joshuamorton 8y ago
I believe that's where the original post's
>given enough time.
comes in.
One nice thing about simulation based research is that it is often (more) reproducible, so a simulation can be run 10000 times, but then the paper might be expected to report how often the simulation succeeded. In other words, you can increase the simulation size to make p-hacking infeasible
Note that in practice, pre-announcing your sample size doesn't prevent p-hacking unless your sample size is == to a known sample. If you say "our sample size will be X", but you can collect 2 or 3x X data even, you can almost certainly p-hack.
Not to mention that I'm unaware of any field where people actually pre-announce their sample sizes. Does this happen on professor's web pages and I'm unaware, or as footnotes in prior papers?
- cepth 8y agoAgain, academia/research is not my profession. But, some cool efforts in this area include osf.io, which is trying to be the Arxiv or Github of preregistration for scientific studies. The best preregistration plans will typically include a declared sample or population to observe (http://datacolada.org/64 http://datacolada.org/64), or at least clear cut criteria for which participants or observations you will exclude. I think for the type of economics/finance research I’m most familiar with, you often implicitly announce your sample when securing funding for a research proposal. E.g. if I’m trying to see if pursuing a momentum strategy with S&P 500 stocks is profitable (a la AQR’s work), it’s pretty obvious what the sample ought to be. This is partly why that meta study I linked to earlier was able to sniff out potential signs of p-hacking.