3 ms·

I agree that the reliance on unpublished and non peer reviewed studies seems to be the biggest flaw, and isn't going to be easily addressed by the author withou

I agree that the reliance on unpublished and non peer reviewed studies seems to be the biggest flaw, and isn't going to be easily addressed by the author without significant work. But one route to prestigious publication might involve publishing each of the independent replications studies as individual papers in a journal that doesn't assess for impact (like PLOS One or PeerJ), then once each has passed peer review on its own and been published write the meta-analysis on those studies and try to publish that analysis in a higher profile journal. It would take a hell of a lot longer, but that's the way to ensure that each individual study passes the bar for scientific rigor, and so therefore your meta-analysis actually has some teeth. And yes, that sounds like a grueling process -- and a damn expensive one if each publication in PLOS One carries a $1,500 fee, so $12,000 just for the author's 8 replication studies. Although this is certainly where PeerJ's unlimited publishing for a one-time fee of $399 would be the cheapest option. [Disclosure: shareholder of SAGE, which is an investor in PeerJ]

mattkrause 10y ago
You wouldn't need to publish 8 papers. The paper is pretty light on details, but there is this table in the appendix: https://osf.io/e6h5d/ https://osf.io/e6h5d/ It looks like there are two paradigms used with different stimulus sets and subject populations. I think a longish paragraph describing each paradigm, plus shorter descriptions of the different subject pools and word lists (and any other manipulation) would be enough.
ramblenode 10y ago
I don't see the need for publishing the constituent studies before publishing the "meta-analysis". The individual studies aren't contributing unique theoretical arguments; they are additive toward a single argument that the original results can't be replicated. It's fairly common for a single paper to incorporate multiple experiments (the paper is closer to this style than a traditional meta-analysis) which are all addressed by the same reviewers.
- dougmccune 10y ago
  Good points and your comment and the other sibling comment definitely make the same valid argument. I got hung up on the phrase meta-analysis, which does seem to be misused, or at least not used in the typical sense (which as was noted in the post was brought up by a reviewer as a problem). It's not a meta-analysis of existing research, it's purely new research to present multiple replications attempts, all of which failed.

mattkrause 10y ago

You wouldn't need to publish 8 papers. The paper is pretty light on details, but there is this table in the appendix: https://osf.io/e6h5d/ https://osf.io/e6h5d/ It looks like there are two paradigms used with different stimulus sets and subject populations. I think a longish paragraph describing each paradigm, plus shorter descriptions of the different subject pools and word lists (and any other manipulation) would be enough.

ramblenode 10y ago

I don't see the need for publishing the constituent studies before publishing the "meta-analysis". The individual studies aren't contributing unique theoretical arguments; they are additive toward a single argument that the original results can't be replicated. It's fairly common for a single paper to incorporate multiple experiments (the paper is closer to this style than a traditional meta-analysis) which are all addressed by the same reviewers.

dougmccune 10y ago
Good points and your comment and the other sibling comment definitely make the same valid argument. I got hung up on the phrase meta-analysis, which does seem to be misused, or at least not used in the typical sense (which as was noted in the post was brought up by a reviewer as a problem). It's not a meta-analysis of existing research, it's purely new research to present multiple replications attempts, all of which failed.