3 ms·

I've read a couple write ups on this situation and I still haven't figured out how they picked the 100 studies. Was it random? It must have been right?

hackuser 11y ago
I had the same question at the time and never found an answer.
harry8 11y ago
Doesn't matter. If read the tea-leaves, listen to rumour or carefully select the 100 papers most likely. The fact you can select 100 papers, do the work to reproduce them and fail in over the half the cases is damning. It shows there exists a systemic problem. Published, peer reviewed, cited work that can't be replicated is a problem. If it were a random sample of papers it is also a systemic problem. Take your pick, it's a systemic problem that needs to be addressed either way.
neilc 11y ago
From the original paper: "We constructed a sampling frame and selection process to minimize selection biases and maximize generalizability of the accumulated evidence. Simultaneously, to maintain high quality, within this sampling frame we matched individual replication projects with teams that had relevant interests and expertise. We pursued a quasi-random sample by defining the sampling frame as 2008 articles of three important psychology journals: Psychological Science (PSCI), Journal of Personality and Social Psychology (JPSP), and Journal of Experimental Psychology: Learning, Memory, and Cognition (JEP: LMC)...The first replication teams could select from a pool of the first 20 articles from each journal, starting with the first article published in the first 2008 issue. Project coordinators facilitated matching articles with replication teams by interests and expertise until the remaining articles were difficult to match. If there were still interested teams, then another 10 articles from one or more of the three journals were made available from the sampling frame. Further, project coordinators actively recruited teams from the community with relevant experience for particular articles."

hackuser 11y ago

I had the same question at the time and never found an answer.

harry8 11y ago

Doesn't matter. If read the tea-leaves, listen to rumour or carefully select the 100 papers most likely. The fact you can select 100 papers, do the work to reproduce them and fail in over the half the cases is damning. It shows there exists a systemic problem. Published, peer reviewed, cited work that can't be replicated is a problem. If it were a random sample of papers it is also a systemic problem. Take your pick, it's a systemic problem that needs to be addressed either way.

neilc 11y ago

From the original paper: "We constructed a sampling frame and selection process to minimize selection biases and maximize generalizability of the accumulated evidence. Simultaneously, to maintain high quality, within this sampling frame we matched individual replication projects with teams that had relevant interests and expertise. We pursued a quasi-random sample by defining the sampling frame as 2008 articles of three important psychology journals: Psychological Science (PSCI), Journal of Personality and Social Psychology (JPSP), and Journal of Experimental Psychology: Learning, Memory, and Cognition (JEP: LMC)...The first replication teams could select from a pool of the first 20 articles from each journal, starting with the first article published in the first 2008 issue. Project coordinators facilitated matching articles with replication teams by interests and expertise until the remaining articles were difficult to match. If there were still interested teams, then another 10 articles from one or more of the three journals were made available from the sampling frame. Further, project coordinators actively recruited teams from the community with relevant experience for particular articles."