4 ms·

My understanding of the paper is less "we couldn't reproduce this paper" and more "insufficient data is available to guarantee any replication is identical". B

My understanding of the paper is less "we couldn't reproduce this paper" and more "insufficient data is available to guarantee any replication is identical". Basically how can you be sure your difference points to a mistake in the original vs a mistake in repetition. Additionally I think it was also a little "replication would be cheaper if we did this". However I think calling it fundamental is a bit of a stress. After all such a paper is going to heavily focus on such details by necessity. You have to categorize to perform such a study and blockers become glaringly obvious. However IMHO I think that a failed replication caused by inaccurate replication isn't a lost cause and instead acts as a new lens to view the original in. "You forgot to mention whether you controlled for X so I did but it didn't work" is valuable but certainly harder to catch in peer review (where the explicit mention might have caused questions). Put another way we can replicate as well as we need to if we prioritize it and while better steps are important they aren't as important as actually replicating. You only once mentioned a single theory which only touches on a part of the replication crisis and not the core issue of "no one can replicate with no funds to do so"...

brutusborn 3y ago
I completely agree that replication is essential, but am arguing that peer review and consensus based science do not help improve replication and thus do not produce better science (generally). We put huge amounts of (unpaid) scientific resources towards peer review and it has hardly improved anything. It’s the opposite: “it’s peer reviewed” has become a reflexive defence of published papers with the incorrect implicit assumption that peer review makes the science better. We should focus on the initiatives that have a proven track record of improving things (like open science) and abandon peer review and reliance on consensus. As you point out, financial limits prevent replication. So why do we spend so much time and money on fancy journals and reviewers who have such a terrible track record of improving things? We could be replicating instead. For a recent example of consensus failures, see Nobel Prize winner Katalin Karikó whos seminal paper was rejected by Nature and whos research programme was rejected by her University for being too incremental. Her peers failed because they relied on flawed heuristics like consensus.
- wolverine876 3y ago
  > We put huge amounts of (unpaid) scientific resources towards peer review and it has hardly improved anything. What was the alternative (and when?), and what were the results? Who will fund all this replication - basically double-funding all research? Do we want our scientists spending their limited time on old things instead of discovering new ones?
  - brutusborn 3y ago
    I have more ideas in my other comments, but fundamentally I think open review is a better norm. Publish a preprint and let the world have at it! You don’t need a committee to decide who your ‘peers’ are: if they care enough they will be reading your paper anyway. Normalising publishing these reviews (like PubPeer) would ensure review, without an arbitrary choice of ‘peers’. Think a paper needs more reviews? Then pay the reviewers and publish their reviews. The benefits of secrecy and gate keeping are negligible compared to the downside. Currently billions are spent on academic publishing even though servers are cheap and the labour is unpaid. This would be a good starting point to fund replications. But that will never happen while the status quo is “peer reviewed = good science” Also, scientists should be spending time on what they think is valuable. Scientists happily replicate important papers using their existing resources. If no-one cares about your result and won't replicate for free, then pay someone to do it! This is the basis of adversarial collaborations (see FEP collaboration funded by a philanthropic foundation). No-one is advocating that e.g. we replicate newtons law for the Nth time. And how to determine if N is enough? Use Lakatos' MSRP to compare rival theories. Again, these dreams are predicated on the status quo changing to where scientists care about the content of a paper, not where is it published or that a faceless committee has accepted its foibles. A world where the public response to a non-replicated result is: "I wonder if any groups plan on replicating?" vs "New paper published in __nature__ has <newsworthy counter-intuitive yet unreplicable result>! Book deal / gov. contract / faculty promotion here I come!"

brutusborn 3y ago

I completely agree that replication is essential, but am arguing that peer review and consensus based science do not help improve replication and thus do not produce better science (generally). We put huge amounts of (unpaid) scientific resources towards peer review and it has hardly improved anything. It’s the opposite: “it’s peer reviewed” has become a reflexive defence of published papers with the incorrect implicit assumption that peer review makes the science better. We should focus on the initiatives that have a proven track record of improving things (like open science) and abandon peer review and reliance on consensus. As you point out, financial limits prevent replication. So why do we spend so much time and money on fancy journals and reviewers who have such a terrible track record of improving things? We could be replicating instead. For a recent example of consensus failures, see Nobel Prize winner Katalin Karikó whos seminal paper was rejected by Nature and whos research programme was rejected by her University for being too incremental. Her peers failed because they relied on flawed heuristics like consensus.

wolverine876 3y ago
> We put huge amounts of (unpaid) scientific resources towards peer review and it has hardly improved anything. What was the alternative (and when?), and what were the results? Who will fund all this replication - basically double-funding all research? Do we want our scientists spending their limited time on old things instead of discovering new ones?
- brutusborn 3y ago
  I have more ideas in my other comments, but fundamentally I think open review is a better norm. Publish a preprint and let the world have at it! You don’t need a committee to decide who your ‘peers’ are: if they care enough they will be reading your paper anyway. Normalising publishing these reviews (like PubPeer) would ensure review, without an arbitrary choice of ‘peers’. Think a paper needs more reviews? Then pay the reviewers and publish their reviews. The benefits of secrecy and gate keeping are negligible compared to the downside. Currently billions are spent on academic publishing even though servers are cheap and the labour is unpaid. This would be a good starting point to fund replications. But that will never happen while the status quo is “peer reviewed = good science” Also, scientists should be spending time on what they think is valuable. Scientists happily replicate important papers using their existing resources. If no-one cares about your result and won't replicate for free, then pay someone to do it! This is the basis of adversarial collaborations (see FEP collaboration funded by a philanthropic foundation). No-one is advocating that e.g. we replicate newtons law for the Nth time. And how to determine if N is enough? Use Lakatos' MSRP to compare rival theories. Again, these dreams are predicated on the status quo changing to where scientists care about the content of a paper, not where is it published or that a faceless committee has accepted its foibles. A world where the public response to a non-replicated result is: "I wonder if any groups plan on replicating?" vs "New paper published in __nature__ has <newsworthy counter-intuitive yet unreplicable result>! Book deal / gov. contract / faculty promotion here I come!"