4 ms·

We can solve this at the grant level. Stipulate that for every new paper a group publishes from a grant, that group must also publish a replication of an existi

by mlsu 2y ago

We can solve this at the grant level. Stipulate that for every new paper a group publishes from a grant, that group must also publish a replication of an existing finding. Publication would happen in pairs, so that every novel thing would be matched with a replication. Replications could be matched with grants: if you receive $100,000 grant, you'd get the $100,000 you need, plus another $100,000 which you could use to publish a replication of a previous $100,000 grant. Researchers can choose which findings they replicate, but with restrictions, e.g. you can't just choose your group's previous thing. I think if we did this, researchers would naturally be incentivized to publish experiments that are easier to replicate and of course fraud like this would be caught eventually. I bet we could throw away half of publications tomorrow and see no effect on the actual pace of progress in science.

scarmig 2y ago
> I bet we could throw away half of publications tomorrow and see no effect on the actual pace of progress in science. It might actually improve the pace of science, if the half eliminated were not replicable and the remaining half were written by researchers knowing that they would likely face a replication attempt.
breuleux 2y ago
There would still be incentives for collusion (I "reproduce" your research, you "reproduce" mine), and researchers pretending to reproduce papers but actually not bothering (especially if they believe that the original research was done properly). Ultimately, I'm not sure how to incentivize reproduction of research: it's very easy to fake a successful reproduction (you already know the results, and the original researcher will not challenge you), so you don't want to reward that too much. Whereas incentivizing failed reproductions might lead some scientists to sabotage their own reproduction efforts in ways that are subtle enough to have plausible deniability. Proceeding by pairs is probably not enough. You probably need 5-6 replications per paper to make sure that at least one attempt is honest and competent, and make the others afraid to do the wrong thing and stand out.
- jimbokun 2y ago
  It is much much harder to sustain a conspiracy among many distributed people over time, than it is to fake your own research results. Making fraud much less convenient will greatly reduce the amount of it.
  - pc86 2y ago
    So does increasing the penalties. If you publish a paper with fraudulent data, methods, or results, and you received any state or federal funds for it, there should be prison time. You stole taxpayer money. I'm not saying for when people are wrong, I'm saying for when you can prove someone knowingly lied. It won't catch anyone, and you need to bar to be high enough that people don't go to jail for being bad scientists, but right now there is zero social, professional, or legal risk is just lying your ass off to get the next grant and keep the spice flowing. Nobody's going to do that when changing the numbers in your Excel sheet carries a risk of a decade or two in a minimum security prison.
- mlsu 2y ago
  You could randomize replications a bit, take away the choice. Or make it so that if you replicated one group's result, you can't replicate them again next time. The key is a bit of distance, a bit of neutrality. Enough jitter to break up cliques. I don't work in academia but in my experience professors are basically all intellectually arrogant and ego-driven, and would relish having time and space to beat each other at the brain game. A failed replication is their chance to be "the smarter guy in the room" and crack open some long-held belief. A successful replication would probably happen most of the time and be far more boring. I could imagine, if such a thing were mandated and in place for a while, one could build her career on replications, as a prosecutor or defense. She would publish new research solely to convince her colleagues that she is sharp enough to play prosecutor or defense. Anything has got to be better than what we have now, where apparently you can cheat and defraud your way through an entire decades-spanning career.
  - robotelvis 2y ago
    The tricky thing with randomizing is that science gets very specialized, both with equipment required and knowledge. So there may only be a handful of people whose work you can competently replicate. And those same people are reviewing the papers you publish and will not hesitate to sabotage your career if you have made them look bad by failing to replicate their papers.
__turbobrew__ 2y ago
That doesn’t fix the incentives, people will just commit fraud when replicating results so they can perform more original research.
- jimbokun 2y ago
  Don't require the replication to be successful. Failure to replicate a result is just as valuable as successfully replicating a result.
  - __turbobrew__ 2y ago
    It is a lot easier to just falsely prove the experiment since the data is already there and the publisher of the paper is not going to push back if you confirm it. Why go through all the work of actually proving/disproving the experiment when you can just change tweak the numbers of the original experiment, say you actually reproduced the experiment, and then move on?
    - KajMagnus 2y ago
      > say you actually reproduced the experiment And get a nice $100k too for close to zero effort! (Having in mind: "plus another $100,000 which you could use to publish a replication")
jltsiren 2y ago
I think it would be better to have separate grants for replication studies. If something becomes a mandatory administrative burden, people will see it as low-prestige work and try to avoid it. And the kind of people who are good at novel research are often also good at ignoring duties they don't like, or completing them with a minimal effort if forced to. But if there is separate funding for replication studies, it will become something people compete for. Some people will specialize on replicating others' work, and universities will pay attention, as they care about grant overheads.
- ubj 2y ago
  > But if there is separate funding for replication studies, it will become something people compete for. It would need to be very good funding on par with what's offered for "novel research". In addition, we would need increased prestige (e.g. awards, citations) for replicated studies as well for this to be effective. For many academics funding is merely a means to that end.
- robotelvis 2y ago
  Another reason for doing this is that if the people doing replication also do original research then calling out someone’s work as bad incentivizes them to sabotage your work when they inevitably review your papers. You can avoid that to some extent by having replication and original work be separate specialities - and making sure that replication gets prestige so good people do it.
mike_hearn 2y ago
Replication is over-emphasised. Attempts to organise mass replications have struggled with basic problems like papers making numerous claims (which one do you replicate?), the question of whether you try to replicate the original methodology exactly or whether you try to answer the same question as the original paper (matters in cases where the methodology was bad), many papers making obvious low value findings (e.g. poor children do worse at school) and so on. But the biggest problem is actually that large swathes of 'scientists' don't do experiments at all. You can't even replicate such papers because they exist purely in the realm of the theoretical. The theory often isn't even properly written down! They will tell you that the paper is just a summary of the real model, which is (at best) found in a giant pile of C or R on some github repo that contains a single commit. Try to replicate their model from the paper, there isn't enough detail to do so. Try to replicate from the code, all you're doing is pointlessly rewriting code that already exists (proves nothing). Try to re-derive their methodology from the original question and if you can't, they'll just reject your paper as illegitimate criticism and say it wasn't a real replication. Having reviewed quite a lot of scientific papers in the past six years or so, the ones that were really problematic couldn't have been fixed with incentivized replication.
- mlsu 2y ago
  So then, how on earth does this stuff even get published? What exactly is it that we're all doing here? If a finding either cannot be communicated enough for someone else to replicate it, or cannot be replicated because the method is shoddy, can we even call that science? At some level I know that what I'm proposing isn't realistic because the majority of science is sloppy. P-hacking, lack of detail, bad writing, bad methods, code that doesn't compile, fraud. But maybe if we tried some version of this, it would cause a course correction. Reviewers, knowing that someone actually would attempt to replicate a paper at some point down the road, would be far more critical of ambiguity and lack of detail. Papers that are not fit to be replicated in the future, whose claims cannot be tested independently, are actually not science at all. They are worth less than nothing because they take up air in the room, choking out actual progress.
  - mike_hearn 2y ago
semiquaver 2y ago
> I bet we could throw away half of publications tomorrow and see no effect on the actual pace of progress in science. Undoubtedly this is true. The problem, like with advertising, is identifying _which_ half to cut.
jimmydorry 2y ago
Would this not incentivise the forming of groups that replicate each others work. If you're already committing wilful fraud on your own papers, why wouldn't you commit a bit more for another researcher willing to do the same for you? With >2 parties, it won't be immediately obvious that this trading has occured.

by mlsu 2y ago

scarmig 2y ago

> I bet we could throw away half of publications tomorrow and see no effect on the actual pace of progress in science. It might actually improve the pace of science, if the half eliminated were not replicable and the remaining half were written by researchers knowing that they would likely face a replication attempt.

breuleux 2y ago

There would still be incentives for collusion (I "reproduce" your research, you "reproduce" mine), and researchers pretending to reproduce papers but actually not bothering (especially if they believe that the original research was done properly). Ultimately, I'm not sure how to incentivize reproduction of research: it's very easy to fake a successful reproduction (you already know the results, and the original researcher will not challenge you), so you don't want to reward that too much. Whereas incentivizing failed reproductions might lead some scientists to sabotage their own reproduction efforts in ways that are subtle enough to have plausible deniability. Proceeding by pairs is probably not enough. You probably need 5-6 replications per paper to make sure that at least one attempt is honest and competent, and make the others afraid to do the wrong thing and stand out.

jimbokun 2y ago
It is much much harder to sustain a conspiracy among many distributed people over time, than it is to fake your own research results. Making fraud much less convenient will greatly reduce the amount of it.
- pc86 2y ago
  So does increasing the penalties. If you publish a paper with fraudulent data, methods, or results, and you received any state or federal funds for it, there should be prison time. You stole taxpayer money. I'm not saying for when people are wrong, I'm saying for when you can prove someone knowingly lied. It won't catch anyone, and you need to bar to be high enough that people don't go to jail for being bad scientists, but right now there is zero social, professional, or legal risk is just lying your ass off to get the next grant and keep the spice flowing. Nobody's going to do that when changing the numbers in your Excel sheet carries a risk of a decade or two in a minimum security prison.
mlsu 2y ago
You could randomize replications a bit, take away the choice. Or make it so that if you replicated one group's result, you can't replicate them again next time. The key is a bit of distance, a bit of neutrality. Enough jitter to break up cliques. I don't work in academia but in my experience professors are basically all intellectually arrogant and ego-driven, and would relish having time and space to beat each other at the brain game. A failed replication is their chance to be "the smarter guy in the room" and crack open some long-held belief. A successful replication would probably happen most of the time and be far more boring. I could imagine, if such a thing were mandated and in place for a while, one could build her career on replications, as a prosecutor or defense. She would publish new research solely to convince her colleagues that she is sharp enough to play prosecutor or defense. Anything has got to be better than what we have now, where apparently you can cheat and defraud your way through an entire decades-spanning career.
- robotelvis 2y ago
  The tricky thing with randomizing is that science gets very specialized, both with equipment required and knowledge. So there may only be a handful of people whose work you can competently replicate. And those same people are reviewing the papers you publish and will not hesitate to sabotage your career if you have made them look bad by failing to replicate their papers.

__turbobrew__ 2y ago

That doesn’t fix the incentives, people will just commit fraud when replicating results so they can perform more original research.

jimbokun 2y ago
Don't require the replication to be successful. Failure to replicate a result is just as valuable as successfully replicating a result.
- __turbobrew__ 2y ago
  It is a lot easier to just falsely prove the experiment since the data is already there and the publisher of the paper is not going to push back if you confirm it. Why go through all the work of actually proving/disproving the experiment when you can just change tweak the numbers of the original experiment, say you actually reproduced the experiment, and then move on?
  - KajMagnus 2y ago
    > say you actually reproduced the experiment And get a nice $100k too for close to zero effort! (Having in mind: "plus another $100,000 which you could use to publish a replication")

jltsiren 2y ago

I think it would be better to have separate grants for replication studies. If something becomes a mandatory administrative burden, people will see it as low-prestige work and try to avoid it. And the kind of people who are good at novel research are often also good at ignoring duties they don't like, or completing them with a minimal effort if forced to. But if there is separate funding for replication studies, it will become something people compete for. Some people will specialize on replicating others' work, and universities will pay attention, as they care about grant overheads.

ubj 2y ago
> But if there is separate funding for replication studies, it will become something people compete for. It would need to be very good funding on par with what's offered for "novel research". In addition, we would need increased prestige (e.g. awards, citations) for replicated studies as well for this to be effective. For many academics funding is merely a means to that end.
robotelvis 2y ago
Another reason for doing this is that if the people doing replication also do original research then calling out someone’s work as bad incentivizes them to sabotage your work when they inevitably review your papers. You can avoid that to some extent by having replication and original work be separate specialities - and making sure that replication gets prestige so good people do it.

mike_hearn 2y ago

Replication is over-emphasised. Attempts to organise mass replications have struggled with basic problems like papers making numerous claims (which one do you replicate?), the question of whether you try to replicate the original methodology exactly or whether you try to answer the same question as the original paper (matters in cases where the methodology was bad), many papers making obvious low value findings (e.g. poor children do worse at school) and so on. But the biggest problem is actually that large swathes of 'scientists' don't do experiments at all. You can't even replicate such papers because they exist purely in the realm of the theoretical. The theory often isn't even properly written down! They will tell you that the paper is just a summary of the real model, which is (at best) found in a giant pile of C or R on some github repo that contains a single commit. Try to replicate their model from the paper, there isn't enough detail to do so. Try to replicate from the code, all you're doing is pointlessly rewriting code that already exists (proves nothing). Try to re-derive their methodology from the original question and if you can't, they'll just reject your paper as illegitimate criticism and say it wasn't a real replication. Having reviewed quite a lot of scientific papers in the past six years or so, the ones that were really problematic couldn't have been fixed with incentivized replication.

mlsu 2y ago
So then, how on earth does this stuff even get published? What exactly is it that we're all doing here? If a finding either cannot be communicated enough for someone else to replicate it, or cannot be replicated because the method is shoddy, can we even call that science? At some level I know that what I'm proposing isn't realistic because the majority of science is sloppy. P-hacking, lack of detail, bad writing, bad methods, code that doesn't compile, fraud. But maybe if we tried some version of this, it would cause a course correction. Reviewers, knowing that someone actually would attempt to replicate a paper at some point down the road, would be far more critical of ambiguity and lack of detail. Papers that are not fit to be replicated in the future, whose claims cannot be tested independently, are actually not science at all. They are worth less than nothing because they take up air in the room, choking out actual progress.
- mike_hearn 2y ago

semiquaver 2y ago

> I bet we could throw away half of publications tomorrow and see no effect on the actual pace of progress in science. Undoubtedly this is true. The problem, like with advertising, is identifying _which_ half to cut.

jimmydorry 2y ago

Would this not incentivise the forming of groups that replicate each others work. If you're already committing wilful fraud on your own papers, why wouldn't you commit a bit more for another researcher willing to do the same for you? With >2 parties, it won't be immediately obvious that this trading has occured.