4 ms·

>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published p

• hoosieree 3y ago
  Half is wildly optimistic.
• advisedwang 3y ago
  It would be more than just half productivity. Not only do you have to do the work twice, but you add the delay of someone else replicating before something can be published and built upon by others. If you are developer, imagine how much your productivity would drop going from a 3 minute build to a 1 day build.
  • orangepurple 3y ago
    Terrible analogy. It might take months to come up with an idea but another should be able to follow your method and implement it much more quickly than it took you to come up with the concept and implement it.
    • magimas 3y ago
      horrible take. Taking the LK99 situation as an example: simply copying and adapting a well described growth recipee to your own setup and lab conditions may take weeks. And how would you address situations where measurement setups only exist once on the earth? How would you do peer replication of LHC measurements? Wait for 50 years till the next super-collider is built and someone else can finally verify the results? On a smaller scale: If you need measurements at a synchrotron radiation source to replicate a measurement, is someone supposed to give up his precious measurement time to replicate a paper he isn't interested in? And is the original author of a paper that's in the queue for peer replication supposed to wait for a year or two till the reviewer gets a beamtime on an appropriate measurement station? Even smaller: I did my PhD in a lab with a specific setup that only a single other group in the world had an equivalent to. You simply would not be able to replicate these results. Peer replication is completely unfeasible in experimental fields of science. The current process of peer review is alright, people just need to learn that single papers standing by themselves don't mean too much. The "peer replication" happens over time anyway when others use the same tools, samples, techniques on related problems and find results in agreement with earlier papers.
    • evandrofisico 3y ago
      Usually coming up with a idea is the easy part. For example, in my PhD project, i started with an idea from my advisor that he had in the early 2000. Implementing the code for the simulation and analysis of the data? four months, at most. Running the simulation? almost three years until I had data with good enough resolution for publishing.
      • tnecniv 3y ago
        It’s also very easy to come up with bad ideas — I did plenty of that and I still do, albeit less than I used to. Finding an idea that is novel, interesting, and tractable given your time, skills, resources, and knowledge of the literature is hard, and maybe the most important skill you develop as a researcher. For a reductive example, the idea to solve P vs NP is a great one, but I’m not going to do that any time soon!
        • PaulHoule 3y ago
          I'd say it is a bad idea because you're not going to succeed at it.
          • tnecniv 3y ago
            Sure because that’s an easy example and kind of my point. When you’re starting your PhD, it can be hard to determine what questions you can and cannot answer given your skills and surrounding literature. That comes from experience and trying (and failing) a lot.
    • cycomanic 3y ago
      I think you don't understand how much work is involved in just building the techniques and expertise to pull some experiments off (let's not even talk about the equipment). Even if someone meticulously documents their process, it could still take months to replicate the results. I'm familiar with lithography/nanofabrication and I know that it is typically the case that a process developed in one clean-room can not be directly applied to a different clean room and instead one has to develop a new process based on what the other results. Even in the same lab it can often happen that if you come back to a process after a longer time, that things don't work out anymore and quite a bit of troubleshooting ensues (maybe a supplier for some chemical changed and even though it should be the same formula it behaves slightly different).
      • RoyalHenOil 3y ago
        Months. Haha. I previously worked in agricultural research (in the private sector), and we spent YEARS trying to replicate some published research from overseas. And that was research that had previously been successfully replicated, and we even flew in the original scientists and borrowed a number of their PhD students for several months, year after year, to help us try to make it work. We never did get it to fully replicate in our country. We ended up having to make some pretty extreme changes to the research to get similar (albeit less reliable) results here. We never did figure out why it worked in one part of the world but not another, since we controlled for every other factor we could think of (including literally importing the original team's lab supplies at great expense, just in case there was some trace contaminant on locally sourced materials).
        • hnfong 3y ago
          > We never did figure out why it worked in one part of the world but not another Doesn't that indicate further research is needed? It sounds fascinating to me. (I know it isn't interesting for the people who couldn't get it working.) It also might indicate that the original research was incomplete in the sense that it might be a fluke due to specific conditions in the original country which isn't universal.
      • faeriechangling 3y ago
  • wilde 3y ago
    You can easily look up non peer reviewed papers on arxiv. Why would this be different with replication?
• ImPostingOnHN 3y ago
  half would only be possible if, for every single paper published by a given team, there exists a second team just as talented as the original team, skilled in that specific package of techniques, just waiting to replicate that paper
• dmarchand90 3y ago
  I believe that, contrary to popular belief, the implementation of this system would lead to a substantial increase in productivity in the long run. Here's why: Currently, a significant proportion of research results in various fields cannot be reproduced. This essentially means that a lot of work turns out to be flawed, leading to wasted efforts (you can refer to the 'reproducibility crisis' for more context). Moreover, future research often builds upon this erroneous information, wasting even more resources. As a result, academic journals get cluttered with substandard work, making them increasingly difficult to monitor and comprehend. Additionally, the overall quality of written communication deteriorates as emphasis shifts from the accurate transfer and reproduction of knowledge to the inflated portrayal of novelty. Now consider a scenario where 50% of all research is dedicated to reproduction. Although this may seem to decelerate progress in the short term, it ensures a more consistent and reliable advancement in the long term. The quality of writing would likely improve to facilitate replication. Furthermore, research methodology would be disseminated more quickly, enhancing overall research effectiveness.
  • matthewdgreen 3y ago
    In the current system scientists allocate reproduction efforts to results that they intend to build on. So if you’ve claimed a breakthrough technique for levitating widgets — and I think this widget technique can be used to build spacecraft (or if I think your technique is wrong) — then I will allocate precious time and resources to reproducing your work. By contrast if I don’t think your work is significant and worth following up on, then I allocate my efforts somewhere else. The advantage is that more apparently-significant results (“might cure cancer”) tend to get a bigger slice of very limited resources, while dead-end or useless results (“might slightly reduce flatulence in cats”) don’t. This distributed entrepreneurial approach isn’t perfect, but it works better than central planning. By contrast you could adopt a Soviet-like approach where cat farts and cancer both share replication resources, but this seems like it would be bad for everyone (except the cats.)
    • GravelRocks 3y ago
      "Cat fart" research might also be incredibly expensive to replicate compared to the "might cure cancer" research. In that case it would effectively get a bigger slice of resources because we're treating all research the same!
• harimau777 3y ago
  The issue that I see is: even if halving productivity is acceptable to the field as a whole; how do you incentivize a given scientist to put in the effort? This seems particularly problematic because it is already notoriously hard to get tenure and academia is already notoriously unrewarding to researchers who don't have tenure.
• mapt 3y ago
  We could easily 10x the funding and 5x the manpower we throw at STEM research if we actually cared what they produced. NSF grants distribute 8.5 billion dollars a year, which is less than Major League Baseball (and its Congressionally granted monopoly) makes. The US Congress has directed 75 billion dollars in aid to Ukraine to date.
  • dkqmduems 3y ago
    That would be reasonable, which isn't the point.
• crote 3y ago
  Where are you going to get the budget to build a second LHC solely for replication? How are you going to replicate a long-term medical cohort study which has been running for thirty years? What about a paper describing a one-off astronomical event, like the "Wow!" signal? What if you research the long-term impact of high-dose radiation exposure during Chernobyl? There is plenty of science out there which financially, practically, or ethically simply by definition cannot be replicated. That doesn't mean their results should not be published. If peer review shows that their methods and analysis are sound, there is no reason to doubt the results.
  • philwelch 3y ago
    I think your examples are more amenable to replication than you assume: > Where are you going to get the budget to build a second LHC solely for replication? In cases like this you could simply have a second, independent team time-sharing the LHC and using it to replicate experiments run by the first team. (And vice- versa). It’s not a perfect replication but it’s probably still an improvement over the “just trust me bro” status quo. > How are you going to replicate a long-term medical cohort study which has been running for thirty years? Run two independent studies in parallel from the beginning. > What about a paper describing a one-off astronomical event, like the "Wow!" signal? There was a ton of effort invested into trying to replicate that observation! Since nobody else ever managed to do so, we can’t draw any conclusions from it. > What if you research the long-term impact of high-dose radiation exposure during Chernobyl? That doesn’t preclude replication unless, for some reason, you’re the only researcher researching the long-term impact of high-dose radiation exposure during Chernobyl.
    • crote 3y ago
      > In cases like this you could simply have a second, independent team time-sharing the LHC and using it to replicate experiments run by the first team. (And vice- versa). It’s not a perfect replication but it’s probably still an improvement over the “just trust me bro” status quo. That's not a true replication, and it isn't going to avoid issues like the OPERA experiment measuring neutrinos going faster than the speed of light due to a loose connector. It would not be any different from having the second team just run their own analysis on the data from the first team - at which point the second team can just as well simply validate the first team's analysis like peer review is currently doing. > Run two independent studies in parallel from the beginning. So all currently-running long-running research has to be thrown out? What if the two studies find very small differences, are you allowed to publish either of them? Are the two teams allowed to collaborate at all? > There was a ton of effort invested into trying to replicate that observation! Since nobody else ever managed to do so, we can’t draw any conclusions from it. You can't "replicate" an observation of a freak astronomical event because you can't trigger a freak astronomical event. At best you can do observations and hope it happens again. We indeed cannot draw any conclusions from it, but that doesn't mean you can't publish papers about it. If replication is mandatory, you would not be allowed to do anything with it at all. > That doesn’t preclude replication unless, for some reason, you’re the only researcher researching the long-term impact of high-dose radiation exposure during Chernobyl. It cannot be reproduced because it would be unethical to expose people to near-fatal levels of radiation simply for reproduction. Simply reusing data from the original test subjects isn't a reproduction, after all.