4 ms·
> "if you fail to replicate, you publish that and show the world that the result shouldn't be believed." A lone report of some observation should never be beli
by nonbel 11y ago
> "if you fail to replicate, you publish that and show the world that the result shouldn't be believed."
A lone report of some observation should never be believed. It should be verified by others who retrace the steps.
>"If you successfully replicate, then it has survived the challenge. The more challenges it survives, the more confidence we can have that the result is valid."
If the observation can be independently replicated, it shows that the methods required are understood well enough to communicate and that it is stable in the face of unknown influences. This increases our confidence that we understand what is going on and that the phenomenon is worth theorizing about.
It has nothing to do with an observation being true or valid. The observation was made, it happened. It is true. It is valid. (Excepting outright fraud, which is treated equivalently to some severely unstable phenomenon)
>"The point of the prediction market here is not to replace independent replication with opinion."
They explicitly say that is the goal in the paper. I like this method of eliciting priors, not that the goal is to substitute it for actual replications.
>"It is to ensure that the energy that gets spent on replication is more likely to be spent effectively."
If a study isn't worth replicating, then it wasn't worth doing and reporting in the first place.
- btilly 11y agoWhile I generally agree with the attitude behind your comments, they do not agree with actual research practice. See http://www.the-scientist.com/?articles.view/articleNo/43875/title/Psychology-s-Failure-to-Replicate/ http://www.the-scientist.com/?articles.view/articleNo/43875/... for horrible replication rates in psychology. See http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0040028 http://journals.plos.org/plosmedicine/article?id=10.1371/jou... for evidence that medicine is no better. See http://www.nature.com/nature/journal/v483/n7391/full/483531a.html http://www.nature.com/nature/journal/v483/n7391/full/483531a... for evidence that the research on which we base clinical drug studies is also in bad shape. At this point anything that focuses attention on, "We should replicate SOMETHING" is a huge improvement. Eventually we should get to, "You should expect to be replicated." But we are a long, loong, loooong ways away from that now.
- nonbel 11y ago>"While I generally agree with the attitude behind your comments, they do not agree with actual research practice." I am well aware of this problem. I expect you will have no more success convincing these people that replication is necessary than you would convincing a fervent religious believer their deity does not exist. These juvenile research practices really need to stop though. It is driving the most intelligent and competent potential contributors away from careers in these areas.