5 ms·

They weren't that careful. It needs to be standard to require "independent" (as far as that is possible) replication of all studies at least once before they ar

by nonbel 8y ago

They weren't that careful. It needs to be standard to require "independent" (as far as that is possible) replication of all studies at least once before they are taken seriously. If that means publishing half the current number of original research papers, that is fine. This simple practice would take care of all sorts of issues (bugs, fraud, bias, random fluctuations).

calvano915 8y ago
And also increase public trust in what is published (less of the "first they said ... was bad for you, now they say ... is good for you", etc)
- nonbel 8y ago
  Well, the way research is usually run they look for "statistical significance" and try to have "power" of ~80% to find it.[1] The power is controlled by the sample size, which is a big determinant of how much is spent on the study so it is chosen to be the lowest acceptable value. Anyway, my point is if there is an 80% chance of getting a result and you run two trials with those same odds, then only about 64% of the studies will replicate with the correct result (in R): p = 0.8 res = t(replicate(1e5, sample(0:1, 2, prob = c(1 - p, p), replace = T))) # 0 = Replicated wrong result # 1 = Non-replicated result # 2 = Replicated correct result > table(rowSums(res)) 0 1 2 3944 32135 63921 So, the standard practice is designed to generate a large percentage of conflicting results. [1] 80% is the target, but is probably usually optimistic
jobigoud 8y ago
How do you independently replicate research that hasn't been published yet? And when it's finally published, do you publish both papers and credit both teams? If not what's the incentive for the replicators to consider a non (yet) influential work?
- nonbel 8y ago
  Both projects are funded simultaneously as part of the same grant. If it is worth doing once, it is worth replicating. Vice versa: If it isn't worth funding the replication then the project isn't worth doing.
  - erichocean 8y ago
    Also, add pre-registration and you've basically fixed the replication crisis. And really, what's better? 400 studies that don't replicate, or 200 that do?
    - nonbel 8y ago
      Obviously pre-registration is required if both groups will follow the same plan. If not enough is known to do that, then there needs to be a pilot study done to figure it out. Basically what is being published now is a bunch of pilot studies.
- thatoneuser 8y ago
  He never said replicate before publish. He said replicate before taking seriously. Of course, it’s not in the hands of the author that they be taken seriously. Essentially what op is saying is the collective community needs to be more scrutinizing, a statement laced with many levels of irony.
pgsandstrom 8y ago
To replicate all published studies is not a simple practice. It would be an enormous task.
- ramblerman 8y ago
  perhaps not all are worth replicating. It might be a net benefit.
thatoneuser 8y ago
Honestly after a group publishes it’s necessarily someone else’s responsibility to reproduce. So publish quickly, and hope it gets enough attention that a reproduction comes quickly. Honestly this seems like more of a media issue. They take a story and run with it and don’t care if it’s right.

by nonbel 8y ago

calvano915 8y ago

And also increase public trust in what is published (less of the "first they said ... was bad for you, now they say ... is good for you", etc)

nonbel 8y ago
Well, the way research is usually run they look for "statistical significance" and try to have "power" of ~80% to find it.[1] The power is controlled by the sample size, which is a big determinant of how much is spent on the study so it is chosen to be the lowest acceptable value. Anyway, my point is if there is an 80% chance of getting a result and you run two trials with those same odds, then only about 64% of the studies will replicate with the correct result (in R): p = 0.8 res = t(replicate(1e5, sample(0:1, 2, prob = c(1 - p, p), replace = T))) # 0 = Replicated wrong result # 1 = Non-replicated result # 2 = Replicated correct result > table(rowSums(res)) 0 1 2 3944 32135 63921 So, the standard practice is designed to generate a large percentage of conflicting results. [1] 80% is the target, but is probably usually optimistic

jobigoud 8y ago

How do you independently replicate research that hasn't been published yet? And when it's finally published, do you publish both papers and credit both teams? If not what's the incentive for the replicators to consider a non (yet) influential work?

nonbel 8y ago
Both projects are funded simultaneously as part of the same grant. If it is worth doing once, it is worth replicating. Vice versa: If it isn't worth funding the replication then the project isn't worth doing.
- erichocean 8y ago
  Also, add pre-registration and you've basically fixed the replication crisis. And really, what's better? 400 studies that don't replicate, or 200 that do?
  - nonbel 8y ago
    Obviously pre-registration is required if both groups will follow the same plan. If not enough is known to do that, then there needs to be a pilot study done to figure it out. Basically what is being published now is a bunch of pilot studies.
thatoneuser 8y ago
He never said replicate before publish. He said replicate before taking seriously. Of course, it’s not in the hands of the author that they be taken seriously. Essentially what op is saying is the collective community needs to be more scrutinizing, a statement laced with many levels of irony.

pgsandstrom 8y ago

To replicate all published studies is not a simple practice. It would be an enormous task.

ramblerman 8y ago
perhaps not all are worth replicating. It might be a net benefit.

thatoneuser 8y ago

Honestly after a group publishes it’s necessarily someone else’s responsibility to reproduce. So publish quickly, and hope it gets enough attention that a reproduction comes quickly. Honestly this seems like more of a media issue. They take a story and run with it and don’t care if it’s right.