5 ms·

Wait, they thought they could replicate 50 papers at the cost of $25k each, and all in the span of one year? Not a chance. I don’t think these guys have any id

by rrock 8y ago

Wait, they thought they could replicate 50 papers at the cost of $25k each, and all in the span of one year? Not a chance. I don’t think these guys have any idea what they’re doing. I feel sorry for the funders.

lsh 8y ago
It's good to have reasonable goals and a budget, even if some of them aren't met. The funders absolutely know what they are doing and handicapping a replication effort because it went a few thousands over budget would be outrageous. There are still unknown quantities at play here, a large scale replication effort hasn't been attempted before and those efforts themselves - the extent of the effort required to replicate and the impact of failure - are also goals. Failure results, of which there have been several already now, can severely damage the reputation of prestigious journals and teams of scientists. They're not doing this because replication is popular, or cheap or because they thought it would be a fun afternoon lark.
LyndsySimon 8y ago
They thought that because they’d already done 100 in another field: https://osf.io/ezcuj/ https://osf.io/ezcuj/ I worked there for three years :)
- lsh 8y ago
  thank you for the link
- rrock 8y ago
  I'm going to guess that psychology research is far less expensive. To give you a sense of perspective here, $25k will get you something like 4 months of a full time employee at the "fresh out of college" experience level, including benefits and research consumables. That also assumes that you have all of the equipment in place. If all there doing is running a few western blots, because that's all they can do on this budget, well then ok. But I'll venture to guess that that's a far cry from reproducing the most important results in these papers.
  - LyndsySimon 8y ago
    As far as I know, the problem wasn't primarily one of underestimating cost or effort - it was one of funding. The non-profit that did this has downsized quite a bit since the RP:CB began and has been seeking other funders.
nonbel 8y ago
They don't try to reproduce all the results in a paper, only a few "important results", more like a spot check. And it is supposed to be much cheaper since the methods have already been figured out...

by rrock 8y ago

lsh 8y ago

It's good to have reasonable goals and a budget, even if some of them aren't met. The funders absolutely know what they are doing and handicapping a replication effort because it went a few thousands over budget would be outrageous. There are still unknown quantities at play here, a large scale replication effort hasn't been attempted before and those efforts themselves - the extent of the effort required to replicate and the impact of failure - are also goals. Failure results, of which there have been several already now, can severely damage the reputation of prestigious journals and teams of scientists. They're not doing this because replication is popular, or cheap or because they thought it would be a fun afternoon lark.

LyndsySimon 8y ago

They thought that because they’d already done 100 in another field: https://osf.io/ezcuj/ https://osf.io/ezcuj/ I worked there for three years :)

lsh 8y ago
thank you for the link
rrock 8y ago
I'm going to guess that psychology research is far less expensive. To give you a sense of perspective here, $25k will get you something like 4 months of a full time employee at the "fresh out of college" experience level, including benefits and research consumables. That also assumes that you have all of the equipment in place. If all there doing is running a few western blots, because that's all they can do on this budget, well then ok. But I'll venture to guess that that's a far cry from reproducing the most important results in these papers.
- LyndsySimon 8y ago
  As far as I know, the problem wasn't primarily one of underestimating cost or effort - it was one of funding. The non-profit that did this has downsized quite a bit since the RP:CB began and has been seeking other funders.

nonbel 8y ago

They don't try to reproduce all the results in a paper, only a few "important results", more like a spot check. And it is supposed to be much cheaper since the methods have already been figured out...