6 ms·

At this point, since papers like Nature and Cell are so important to scientists, could it be feasible for them to simply require any submitted paper to them wou

by tinco 10y ago

At this point, since papers like Nature and Cell are so important to scientists, could it be feasible for them to simply require any submitted paper to them would only qualify for publication if the results were independently replicated? They could even smoothen the process by giving the draft a 'Nature seal of approval' that the authors could use to get other institutions to replicate their work, and add a small 'Replicated by XX' badge to each publication to reward any institution that replicated a study. Funders of studies might improve the quality of the research they paid for by offering replication rewards. I.e. 5% of all funding goes towards institutions who replicated results from research they funded. Of course there would still be some wrinkles to iron out, but surely we could come up with a nicely balanced solution?

virusduck 10y ago
I think you vastly underestimate how much a lot of scientific studies cost. It would be ideal to be able to have studies replicated by a separate group, but there are many times labs that have specialized equipment and engineering that make replicating studies by some random group unfeasible. In addition, if you spend too much time trying to replicate others' work, you have no time to work on the things that will actually continue to fund your own research. The best thing is to have a healthy, skeptical, but collegial, competition in the field. That still requires more funding though!
- mixedCase 10y ago
  > I think you vastly underestimate how much a lot of scientific studies cost. On the other hand, the money currently being invested seems to produce mostly garbage. "Fake science", just to rustle some feathers.
- RodericDay 10y ago
  > I think you vastly underestimate how much a lot of scientific studies cost Do you have data for this claim? There's tons of extraordinarily expensive experiments through and through, but there's also stuff with incredibly high and time-consuming up-front design and exploration costs that is actually almost trivial to replicate on a per-unit basis.
- Tsagadai 10y ago
  If it only works on a Wizbang 200 and cannot be reproduced with the Doodad 3000 then it probably doesn't work or there are other variables missing from their results. Specialised equipment should not affect results (proving that equipment is broken has some value though).
- Fomite 10y ago
  Really, even without specialized equipment, science is expensive. And many scientists got into the field because they wanted to do their own work. Locking a lab that can't afford to fund an independent replication study out, or who isn't prominent enough to be able to rouse someone's attention to do it, would be disastrous for a lot of early career researchers.
graeham 10y ago
This would be a pretty big step - right now there is a big incentive to be first to publish. Often the researchers who would be doing the replication are competitors, who are racing to publish their results first. Part of publishing is to put it out for discussion. It should be the best effort and thought to be true, but science is about the evolution of knowledge. Very few things are completely understood, especially in biological and micro/nano fields.
dekhn 10y ago
I published in nature. The results were based on a computation that was so large, no other entity in the world could have reproduced it at the time (I work for Google, we built a system that used 600+Kcores continuously for years). I don't think ,at the time, that anybody other than Amazon or Microsoft could have run the same calculation as us. Should our publication have been prevented simply because nobody else had the resources to replicate?
- seszett 10y ago
  > Should our publication have been prevented simply because nobody else had the resources to replicate? Well... maybe, yes? Just because you are the only ones able to do the computation doesn't mean your methods or your results are correct.
  - dekhn 10y ago
    We share all the results from our research, so people may inspect them in detail. We also share our methods in enough detail that if somebody was sufficiently motivated, they could reproduce the work by spending time and money however, If you follow your logic, you would prevent CERN from publishing the discovery of the Higgs Boson, because nobody else has a particle accelerator powerful enough to detect them. You would prevent the LIGO people from publishing because nobody else has a multi-mile-long interferometer that can detect gravity waves. There are many unique resources that would be prevented from publishing under your idea. Contrary to what most non-scientists claim about science, reproducibility is not a requirement for discovery science. What is import is scrutiny- ensuring that when people discover something using a unique resource, other people can inspect the methods and the data and decide whether to believe the results.
    - tinco 10y ago
      Though I agree for the most part, isn't CERN using multiple teams on multiple detectors, even going so far as having multiple implementations of data analyses and such, just to make sure any result they find is never a product of one person or team making a mistake in their process?
deleted 10y ago
[deleted]
Fomite 10y ago
This would effectively lock certain fields out of publishing in those journals. For example, a great deal of population health research involves following people for huge spans of time.

by tinco 10y ago

virusduck 10y ago

I think you vastly underestimate how much a lot of scientific studies cost. It would be ideal to be able to have studies replicated by a separate group, but there are many times labs that have specialized equipment and engineering that make replicating studies by some random group unfeasible. In addition, if you spend too much time trying to replicate others' work, you have no time to work on the things that will actually continue to fund your own research. The best thing is to have a healthy, skeptical, but collegial, competition in the field. That still requires more funding though!

mixedCase 10y ago
> I think you vastly underestimate how much a lot of scientific studies cost. On the other hand, the money currently being invested seems to produce mostly garbage. "Fake science", just to rustle some feathers.
RodericDay 10y ago
> I think you vastly underestimate how much a lot of scientific studies cost Do you have data for this claim? There's tons of extraordinarily expensive experiments through and through, but there's also stuff with incredibly high and time-consuming up-front design and exploration costs that is actually almost trivial to replicate on a per-unit basis.
Tsagadai 10y ago
If it only works on a Wizbang 200 and cannot be reproduced with the Doodad 3000 then it probably doesn't work or there are other variables missing from their results. Specialised equipment should not affect results (proving that equipment is broken has some value though).
Fomite 10y ago
Really, even without specialized equipment, science is expensive. And many scientists got into the field because they wanted to do their own work. Locking a lab that can't afford to fund an independent replication study out, or who isn't prominent enough to be able to rouse someone's attention to do it, would be disastrous for a lot of early career researchers.

graeham 10y ago

This would be a pretty big step - right now there is a big incentive to be first to publish. Often the researchers who would be doing the replication are competitors, who are racing to publish their results first. Part of publishing is to put it out for discussion. It should be the best effort and thought to be true, but science is about the evolution of knowledge. Very few things are completely understood, especially in biological and micro/nano fields.

dekhn 10y ago

I published in nature. The results were based on a computation that was so large, no other entity in the world could have reproduced it at the time (I work for Google, we built a system that used 600+Kcores continuously for years). I don't think ,at the time, that anybody other than Amazon or Microsoft could have run the same calculation as us. Should our publication have been prevented simply because nobody else had the resources to replicate?

seszett 10y ago
> Should our publication have been prevented simply because nobody else had the resources to replicate? Well... maybe, yes? Just because you are the only ones able to do the computation doesn't mean your methods or your results are correct.
- dekhn 10y ago
  We share all the results from our research, so people may inspect them in detail. We also share our methods in enough detail that if somebody was sufficiently motivated, they could reproduce the work by spending time and money however, If you follow your logic, you would prevent CERN from publishing the discovery of the Higgs Boson, because nobody else has a particle accelerator powerful enough to detect them. You would prevent the LIGO people from publishing because nobody else has a multi-mile-long interferometer that can detect gravity waves. There are many unique resources that would be prevented from publishing under your idea. Contrary to what most non-scientists claim about science, reproducibility is not a requirement for discovery science. What is import is scrutiny- ensuring that when people discover something using a unique resource, other people can inspect the methods and the data and decide whether to believe the results.
  - tinco 10y ago
    Though I agree for the most part, isn't CERN using multiple teams on multiple detectors, even going so far as having multiple implementations of data analyses and such, just to make sure any result they find is never a product of one person or team making a mistake in their process?

deleted 10y ago

[deleted]

Fomite 10y ago

This would effectively lock certain fields out of publishing in those journals. For example, a great deal of population health research involves following people for huge spans of time.