4 ms·

The paper was a meta-analysis of nine unpublished studies. While the issue of non-publication of negative results is extremely important, this should not be the

The paper was a meta-analysis of nine unpublished studies. While the issue of non-publication of negative results is extremely important, this should not be the test case for it. I agree with other comments that the lack of nul-results in journals is a massive showstopper for epistemology and confidence in scientific work. In my experience during my PhD, academics were unwilling to submit nuls, they wanted to find the actual answer and publish that instead - which leads to delays of years, leaving well known and obvious errors in existing papers unchallenged in public, and potentially leading to that scientist never even submitting the nul result.

rtpg 10y ago
Would it be possible to take the studies and actually develop them into the analysis? > the authors focus on a bunch of unpublished studies from a dissertation and a colleague who is not even an author of the present paper. There is no way of knowing whether these unpublished experiments meet the standards to be published in high-quality journals.” The way to know if the experiments meet the standards is to show it in the same paper, I guess?
- mattkrause 10y ago
  That's what I would do. I think that isn't actually an unreasonable complaint: the current paper has very few details about the paradigm used in those experiments. There is a really brief table, along with some comments embedded in R output, but what the paper really needs is something that looks like a methods section from an experimental psycholinguistics paper. Something like this: Experiment 1: Nineteen native English speakers (demographics here) performed a lexical decision task. Three string of letters was presented on a computer monitor (viewing conditions here). Subjects were asked to report, via a button press, whether the middle string was a valid English word (left button) or psuedoword (right button). Although they had up to 5 sec to respond, subjects were instructed to respond as quickly as possible and the median reaction time was N.NN seconds. The original experiment manipulated (some irrelevant condition), which is not analyzed here. Instead, we calculated... Experiment 2: As above, except the subject pool consisted of native German speakers and the stimuli were either German words or German psuedowords. Experiment 3: Same as experiment 1, except that (irrelevant condition #2) was also manipulated instead of (irrelevant condition #1). This might not fit in the main paper--even online-only journals can be weird about page limits--but it would be fine in an appendix.
stdbrouw 10y ago
> The paper was a meta-analysis of nine unpublished studies. While the issue of non-publication of negative results is extremely important, this should not be the test case for it. I think they probably made a mistake in calling it a meta-analysis, because indeed a meta-analysis relies on the idea that the studies themselves have already been peer reviewed and as such don't need to be questioned individually. Here, however, we're dealing with eight attempts at replication by the author that have not previously been published, and as such those studies should be discussed in detail so we can assess the merits of the individual replication attempts. Any meta-analysis of those experiments is then really just the cherry on top.
- cwingrav 10y ago
  > the studies themselves have already been peer reviewed and as such don't need to be questioned individually Just to add a bit to your comment: Peer review is a community's way of saying a paper stands up to their best understanding of methods at the time. The author could have accidentally left something out, misrepresented their approach or fabricated things. Peer review would not catch that, but replication or failure to replicate would. They are two different things. For example, I remember one experiment could never be replicated by anyone except the author and his students. So, it got a bit heated. Ultimately, they found it was an experimental setup, not reported in the paper because they didn't think it was important, that allowed the experiment to work. So, it passed peer review, failed replication, but ultimately, because of failed replications and the academic process, further information came to light and knowledge was created. The academic process can be slow and frustrating. But, when applied properly, it is our best approach to expanding knowledge. That isn't to say it can't be improved, especially regarding failure to replicate and null-hypothesis papers. Academics know this and are working on methods around this, methods called out in these discussions. Some of the most innovative people i've found, seeking ways to improve this process have been NSF and NIH officials. So, it's promising, but slow.
  - mattkrause 10y ago
    I think you're putting a little too much weight on peer review here. The way it is presented to non-academics often makes it sound more like a trial or inquest, where a large group of people carefully weigh the evidence. In practice, peer review means that 1-3 people each spent 1-3 hours thinking about the paper and didn't find anything horribly wrong with it. It is more like a code review--it's good if it catches something, but not terribly surprising if some bugs slip through. You're definitely right about the the importance of continuing to discuss and replicate work after it has been published.
dougmccune 10y ago
I agree that the reliance on unpublished and non peer reviewed studies seems to be the biggest flaw, and isn't going to be easily addressed by the author without significant work. But one route to prestigious publication might involve publishing each of the independent replications studies as individual papers in a journal that doesn't assess for impact (like PLOS One or PeerJ), then once each has passed peer review on its own and been published write the meta-analysis on those studies and try to publish that analysis in a higher profile journal. It would take a hell of a lot longer, but that's the way to ensure that each individual study passes the bar for scientific rigor, and so therefore your meta-analysis actually has some teeth. And yes, that sounds like a grueling process -- and a damn expensive one if each publication in PLOS One carries a $1,500 fee, so $12,000 just for the author's 8 replication studies. Although this is certainly where PeerJ's unlimited publishing for a one-time fee of $399 would be the cheapest option. [Disclosure: shareholder of SAGE, which is an investor in PeerJ]
- mattkrause 10y ago
  You wouldn't need to publish 8 papers. The paper is pretty light on details, but there is this table in the appendix: https://osf.io/e6h5d/ https://osf.io/e6h5d/ It looks like there are two paradigms used with different stimulus sets and subject populations. I think a longish paragraph describing each paradigm, plus shorter descriptions of the different subject pools and word lists (and any other manipulation) would be enough.
- ramblenode 10y ago
  I don't see the need for publishing the constituent studies before publishing the "meta-analysis". The individual studies aren't contributing unique theoretical arguments; they are additive toward a single argument that the original results can't be replicated. It's fairly common for a single paper to incorporate multiple experiments (the paper is closer to this style than a traditional meta-analysis) which are all addressed by the same reviewers.
  - dougmccune 10y ago
    Good points and your comment and the other sibling comment definitely make the same valid argument. I got hung up on the phrase meta-analysis, which does seem to be misused, or at least not used in the typical sense (which as was noted in the post was brought up by a reviewer as a problem). It's not a meta-analysis of existing research, it's purely new research to present multiple replications attempts, all of which failed.
deleted 10y ago
[deleted]
georgeecollins 10y ago
You are probably right about the flaws, but my concern is that the bar to publish a study that reports "failure to replicate" is higher. This is at the heart of a very serious problem with science these days. Proving that research can be replicated and validated is probably more socially useful in some fields than "discovering" some new correlation. But all the academic incentives point to publishing "discovery".

rtpg 10y ago

Would it be possible to take the studies and actually develop them into the analysis? > the authors focus on a bunch of unpublished studies from a dissertation and a colleague who is not even an author of the present paper. There is no way of knowing whether these unpublished experiments meet the standards to be published in high-quality journals.” The way to know if the experiments meet the standards is to show it in the same paper, I guess?

mattkrause 10y ago
That's what I would do. I think that isn't actually an unreasonable complaint: the current paper has very few details about the paradigm used in those experiments. There is a really brief table, along with some comments embedded in R output, but what the paper really needs is something that looks like a methods section from an experimental psycholinguistics paper. Something like this: Experiment 1: Nineteen native English speakers (demographics here) performed a lexical decision task. Three string of letters was presented on a computer monitor (viewing conditions here). Subjects were asked to report, via a button press, whether the middle string was a valid English word (left button) or psuedoword (right button). Although they had up to 5 sec to respond, subjects were instructed to respond as quickly as possible and the median reaction time was N.NN seconds. The original experiment manipulated (some irrelevant condition), which is not analyzed here. Instead, we calculated... Experiment 2: As above, except the subject pool consisted of native German speakers and the stimuli were either German words or German psuedowords. Experiment 3: Same as experiment 1, except that (irrelevant condition #2) was also manipulated instead of (irrelevant condition #1). This might not fit in the main paper--even online-only journals can be weird about page limits--but it would be fine in an appendix.

stdbrouw 10y ago

> The paper was a meta-analysis of nine unpublished studies. While the issue of non-publication of negative results is extremely important, this should not be the test case for it. I think they probably made a mistake in calling it a meta-analysis, because indeed a meta-analysis relies on the idea that the studies themselves have already been peer reviewed and as such don't need to be questioned individually. Here, however, we're dealing with eight attempts at replication by the author that have not previously been published, and as such those studies should be discussed in detail so we can assess the merits of the individual replication attempts. Any meta-analysis of those experiments is then really just the cherry on top.

cwingrav 10y ago
> the studies themselves have already been peer reviewed and as such don't need to be questioned individually Just to add a bit to your comment: Peer review is a community's way of saying a paper stands up to their best understanding of methods at the time. The author could have accidentally left something out, misrepresented their approach or fabricated things. Peer review would not catch that, but replication or failure to replicate would. They are two different things. For example, I remember one experiment could never be replicated by anyone except the author and his students. So, it got a bit heated. Ultimately, they found it was an experimental setup, not reported in the paper because they didn't think it was important, that allowed the experiment to work. So, it passed peer review, failed replication, but ultimately, because of failed replications and the academic process, further information came to light and knowledge was created. The academic process can be slow and frustrating. But, when applied properly, it is our best approach to expanding knowledge. That isn't to say it can't be improved, especially regarding failure to replicate and null-hypothesis papers. Academics know this and are working on methods around this, methods called out in these discussions. Some of the most innovative people i've found, seeking ways to improve this process have been NSF and NIH officials. So, it's promising, but slow.
- mattkrause 10y ago
  I think you're putting a little too much weight on peer review here. The way it is presented to non-academics often makes it sound more like a trial or inquest, where a large group of people carefully weigh the evidence. In practice, peer review means that 1-3 people each spent 1-3 hours thinking about the paper and didn't find anything horribly wrong with it. It is more like a code review--it's good if it catches something, but not terribly surprising if some bugs slip through. You're definitely right about the the importance of continuing to discuss and replicate work after it has been published.

dougmccune 10y ago

I agree that the reliance on unpublished and non peer reviewed studies seems to be the biggest flaw, and isn't going to be easily addressed by the author without significant work. But one route to prestigious publication might involve publishing each of the independent replications studies as individual papers in a journal that doesn't assess for impact (like PLOS One or PeerJ), then once each has passed peer review on its own and been published write the meta-analysis on those studies and try to publish that analysis in a higher profile journal. It would take a hell of a lot longer, but that's the way to ensure that each individual study passes the bar for scientific rigor, and so therefore your meta-analysis actually has some teeth. And yes, that sounds like a grueling process -- and a damn expensive one if each publication in PLOS One carries a $1,500 fee, so $12,000 just for the author's 8 replication studies. Although this is certainly where PeerJ's unlimited publishing for a one-time fee of $399 would be the cheapest option. [Disclosure: shareholder of SAGE, which is an investor in PeerJ]

mattkrause 10y ago
You wouldn't need to publish 8 papers. The paper is pretty light on details, but there is this table in the appendix: https://osf.io/e6h5d/ https://osf.io/e6h5d/ It looks like there are two paradigms used with different stimulus sets and subject populations. I think a longish paragraph describing each paradigm, plus shorter descriptions of the different subject pools and word lists (and any other manipulation) would be enough.
ramblenode 10y ago
I don't see the need for publishing the constituent studies before publishing the "meta-analysis". The individual studies aren't contributing unique theoretical arguments; they are additive toward a single argument that the original results can't be replicated. It's fairly common for a single paper to incorporate multiple experiments (the paper is closer to this style than a traditional meta-analysis) which are all addressed by the same reviewers.
- dougmccune 10y ago
  Good points and your comment and the other sibling comment definitely make the same valid argument. I got hung up on the phrase meta-analysis, which does seem to be misused, or at least not used in the typical sense (which as was noted in the post was brought up by a reviewer as a problem). It's not a meta-analysis of existing research, it's purely new research to present multiple replications attempts, all of which failed.

deleted 10y ago

[deleted]

georgeecollins 10y ago

You are probably right about the flaws, but my concern is that the bar to publish a study that reports "failure to replicate" is higher. This is at the heart of a very serious problem with science these days. Proving that research can be replicated and validated is probably more socially useful in some fields than "discovering" some new correlation. But all the academic incentives point to publishing "discovery".