4 ms·

It certainly is counterproductive. We should be incentivising replication, not novelty. If you started giving out grants aimed at replicating results we would s

It certainly is counterproductive. We should be incentivising replication, not novelty. If you started giving out grants aimed at replicating results we would speed up scientific progress ten fold.

auggierose 6y ago
I don't think that is the answer either. We would just get a lot of replications of boring results. And replications of those replications.
- st1x7 6y ago
  That's good. Replication is valuable for science. Boring is fine. Thinking that boring results aren't useful is exactly how we ended up with the current problems in research. You just need to make sure that you don't give scientists an incentive to redo studies that have been replicated so much that additional replication is useless.
  - auggierose 6y ago
    "just"? That's exactly the problem I was referring to.
    - st1x7 6y ago
      That's fair. I wanted to point out that the answer is more complex than "don't incentivise replication".
    - malwarebytess 6y ago
      And he's pointing out that your problem is not really so big so as to allow a casual dismissal of the solution. You've reduced the solution to absurdity by creating a scenario that is at least on face easily remediable.
      - auggierose 6y ago
        > that is at least on face easily remediable But it's not easily remediable. And that is the point. Just imagine how that would work in practice. Somebody does an original experiment that gives exciting results. Let's say they get a 1000 Science Points for that. Now somebody replicates that experiment. How many points should they get? Should you get as many points for replicating a 1000 points experiment as replicating a 500 points experiment? Why would you do original experiments at all anymore? Isn't it easier just to replicate original experiments where all of the hard work has already been done? Awarding points for replication is a nonsense idea.
        OnlyOneCannolo 6y ago
        Giving science points to anyone for every replication they do is one idea. Another way to encourage replication without detracting from original research might be to encourage doing one replication. Maybe something like getting masters students to do them as their thesis or final project.
        emerongi 6y ago
        Well, you wouldn't get 1000 science points for it before it's replicated by others. Once it's replicated enough times by credible people, the points are awarded, and the process is now considered "done". If someone replicates it after this process and finds different results, it's "new research" again and needs to be replicated again.
        carlmr 6y ago
        I agree, it's kind of like people handing in their "finished" feature with "just the tests" missing. It's not finished. It maybe doesn't even work. Simple as that. I think we shouldn't even accept papers that haven't been replicated twice by independent teams.
        auggierose 6y ago
        Doesn't really matter WHEN you get the points for your original experiment. My argument stays the same. Honest researchers doing original research to the best of their abilities. That's how you get good results, and what drives progress. The rest is just bureaucracy. The need for replication will be just another bureaucratic add-on to catch dishonest researchers, and researchers who value prestige more than the truth.
- pa7x1 6y ago
  Why do you assume that the reward for replication needs to be a constant?
  - auggierose 6y ago
    ? I don't.
glial 6y ago
Replicating studies seems like the perfect training exercise for graduate students. I wish it were a mandatory part of training at most schools.
- yhoneycomb 6y ago
  The problem is... if you can't replicate the experiment, how do you know if it's the graduate student's fault or the experiments fault?
  - blackbear_ 6y ago
    The more people fail, the more likely that the fault is on the experimenters' side
  - adamnemecek 6y ago
    If it has been replicated many times then you know whose fault it is.
  - leetcrew 6y ago
    you could ask the same question of someone with a phd.
    - netizen-9748 6y ago
      This is a very good point, we are all still fallible humans no matter what degree w hold
  - glial 6y ago
    Publish methods and results in a database. Every result will be a draw from a distribution. Today only exciting ones get published but it would be better to see the full distribution.
  - rflrob 6y ago
    I always thought that if I ran my own lab, I would have rotation projects be to replicate a recent result from the lab. Then, a lot of the infrastructure and support would still be there, so it would be pretty clear if the fault lay with the experiment. Plus, it would reinforce the notion among trainees that the point of science is to be replicated, and that the hard part of doing something novel is figuring out what to do, not actually doing the work.

It certainly is counterproductive. We should be incentivising replication, not novelty. If you started giving out grants aimed at replicating results we would speed up scientific progress ten fold.

auggierose 6y ago

I don't think that is the answer either. We would just get a lot of replications of boring results. And replications of those replications.

st1x7 6y ago
That's good. Replication is valuable for science. Boring is fine. Thinking that boring results aren't useful is exactly how we ended up with the current problems in research. You just need to make sure that you don't give scientists an incentive to redo studies that have been replicated so much that additional replication is useless.
- auggierose 6y ago
  "just"? That's exactly the problem I was referring to.
  - st1x7 6y ago
    That's fair. I wanted to point out that the answer is more complex than "don't incentivise replication".
  - malwarebytess 6y ago
    And he's pointing out that your problem is not really so big so as to allow a casual dismissal of the solution. You've reduced the solution to absurdity by creating a scenario that is at least on face easily remediable.
    - auggierose 6y ago
      > that is at least on face easily remediable But it's not easily remediable. And that is the point. Just imagine how that would work in practice. Somebody does an original experiment that gives exciting results. Let's say they get a 1000 Science Points for that. Now somebody replicates that experiment. How many points should they get? Should you get as many points for replicating a 1000 points experiment as replicating a 500 points experiment? Why would you do original experiments at all anymore? Isn't it easier just to replicate original experiments where all of the hard work has already been done? Awarding points for replication is a nonsense idea.
      - OnlyOneCannolo 6y ago
        Giving science points to anyone for every replication they do is one idea. Another way to encourage replication without detracting from original research might be to encourage doing one replication. Maybe something like getting masters students to do them as their thesis or final project.
      - emerongi 6y ago
        Well, you wouldn't get 1000 science points for it before it's replicated by others. Once it's replicated enough times by credible people, the points are awarded, and the process is now considered "done". If someone replicates it after this process and finds different results, it's "new research" again and needs to be replicated again.
        carlmr 6y ago
        I agree, it's kind of like people handing in their "finished" feature with "just the tests" missing. It's not finished. It maybe doesn't even work. Simple as that. I think we shouldn't even accept papers that haven't been replicated twice by independent teams.
        auggierose 6y ago
        Doesn't really matter WHEN you get the points for your original experiment. My argument stays the same. Honest researchers doing original research to the best of their abilities. That's how you get good results, and what drives progress. The rest is just bureaucracy. The need for replication will be just another bureaucratic add-on to catch dishonest researchers, and researchers who value prestige more than the truth.
pa7x1 6y ago
Why do you assume that the reward for replication needs to be a constant?
- auggierose 6y ago
  ? I don't.

glial 6y ago

Replicating studies seems like the perfect training exercise for graduate students. I wish it were a mandatory part of training at most schools.

yhoneycomb 6y ago
The problem is... if you can't replicate the experiment, how do you know if it's the graduate student's fault or the experiments fault?
- blackbear_ 6y ago
  The more people fail, the more likely that the fault is on the experimenters' side
- adamnemecek 6y ago
  If it has been replicated many times then you know whose fault it is.
- leetcrew 6y ago
  you could ask the same question of someone with a phd.
  - netizen-9748 6y ago
    This is a very good point, we are all still fallible humans no matter what degree w hold
- glial 6y ago
  Publish methods and results in a database. Every result will be a draw from a distribution. Today only exciting ones get published but it would be better to see the full distribution.
- rflrob 6y ago
  I always thought that if I ran my own lab, I would have rotation projects be to replicate a recent result from the lab. Then, a lot of the infrastructure and support would still be there, so it would be pretty clear if the fault lay with the experiment. Plus, it would reinforce the notion among trainees that the point of science is to be replicated, and that the hard part of doing something novel is figuring out what to do, not actually doing the work.