4 ms·

While I agree with Kahneman that it is easy to fool yourself by looking at studies with small sample size, the recent focus on reproducing existing research is,

While I agree with Kahneman that it is easy to fool yourself by looking at studies with small sample size, the recent focus on reproducing existing research is, to my mind, misguided. It is extremely difficult to perform experiments exactly the same, even within one lab, much less between many different labs. For a good example of this, see this fascinating description of how several labs tried to standardize the way they handle c. elegans (worms) to identify compounds which extended the life of the animals: http://www.nature.com/news/a-long-journey-to-reproducible-results-1.22478 http://www.nature.com/news/a-long-journey-to-reproducible-re... While some effort to standardize is important, it also waste a lot of time setting up a specific set of experimental conditions that may not have much resemblance to the conditions that obtain in the real world. In my opinion, we learn much more by taking someone's existing result, thinking through the consequences and then designing well-powered experiments that probe the assumptions, mechanisms and applicability of the result. With critical eyes and diverse systems, we won't fool ourselves. One more note: if this topic interests you, please read The Structure of Scientific Revolutions. If you are unfamiliar with the book, I guarantee it will completely change how you think about science as a human endeavour and make you much more comfortable with the existence of long periods of time where science just gets some things wrong.

nonbel 9y ago
>"Hundreds of e-mails and many teleconferences later, we converged on a technique but still had a stupendous three-day difference in lifespan between labs. The problem, it turned out, was notation — one lab determined age on the basis of when an egg hatched, others on when it was laid." Wow... this is pretty basic info on what was measured.
Strilanc 9y ago
I think it's important to walk before you run. If social scientists aren't able to get compatible results from the same experiment done by two different teams, then I seriously doubt their ability to extract any relevant subtle truths about human behavior. The c. elegans story you linked is a great example of this. Getting the little details right matters. I can't imagine having any luck doing that while constantly changing the big details.
ianamartin 9y ago
This is another vote for The Structure of Scientific Revolutions. Such an incredible book.
Vinnl 9y ago
> In my opinion, we learn much more by taking someone's existing result, thinking through the consequences and then designing well-powered experiments that probe the assumptions, mechanisms and applicability of the result. I think that, too, is considered replication by many.
- rgejman 9y ago
  The replication projects I am specifically referring to are similar to eLife's "Reproducibility Project: Cancer Biology": https://elifesciences.org/collections/9b1e83d1/reproducibility-project-cancer-biology https://elifesciences.org/collections/9b1e83d1/reproducibili... http://www.sciencemag.org/news/2017/01/rigorous-replication-effort-succeeds-just-two-five-cancer-papers http://www.sciencemag.org/news/2017/01/rigorous-replication-...

nonbel 9y ago

>"Hundreds of e-mails and many teleconferences later, we converged on a technique but still had a stupendous three-day difference in lifespan between labs. The problem, it turned out, was notation — one lab determined age on the basis of when an egg hatched, others on when it was laid." Wow... this is pretty basic info on what was measured.

Strilanc 9y ago

I think it's important to walk before you run. If social scientists aren't able to get compatible results from the same experiment done by two different teams, then I seriously doubt their ability to extract any relevant subtle truths about human behavior. The c. elegans story you linked is a great example of this. Getting the little details right matters. I can't imagine having any luck doing that while constantly changing the big details.

ianamartin 9y ago

This is another vote for The Structure of Scientific Revolutions. Such an incredible book.

Vinnl 9y ago

> In my opinion, we learn much more by taking someone's existing result, thinking through the consequences and then designing well-powered experiments that probe the assumptions, mechanisms and applicability of the result. I think that, too, is considered replication by many.

rgejman 9y ago
The replication projects I am specifically referring to are similar to eLife's "Reproducibility Project: Cancer Biology": https://elifesciences.org/collections/9b1e83d1/reproducibility-project-cancer-biology https://elifesciences.org/collections/9b1e83d1/reproducibili... http://www.sciencemag.org/news/2017/01/rigorous-replication-effort-succeeds-just-two-five-cancer-papers http://www.sciencemag.org/news/2017/01/rigorous-replication-...