13 ms·

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published pa

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of the preparation that way. But properly replicating a paper will still be a lot of effort, especially when there are any issues and it doesn't work on the first try. Then you have to troubleshoot your experiments and make sure that no mistakes were made. That can add a lot of time to the process. This is also all work that doesn't benefit the scientists replicating the paper. It only costs them money and time. If someone cares enough about the work to build on it, they will replicate it anyway. And in that case they have a good incentive to spend the effort. If that works this will indirectly support the original paper even if the following papers don't specifically replicate the original results. Though this part is much more problematic if the following experiments fail, then this will likely remain entirely unpublished. But the solution here unfortunately isn't as simple as just publishing negative results, it take far more work to create a solid negative result than just trying the experiments and abandoning them if they're not promising.

backtoyoujim 3y ago
Yes it would indeed mean slowing down and having more scientists. It would mean disruption is no longer a useful tool for human development.
- brnaftr361 3y ago
  It may not be. I would be willing to argue that there was a tipping point and we've long exceeded its boundary - progress and disruption now is just making finding an equilibrium in the future increasingly difficult. So entering into a paradigm where we test the known space - especially presently - would 1) help reduce cruft; 2) abate undersirable forward progress; 3) train the next generation(s) of scientists to be more diligent and better custodians of the domain.
- ebiester 3y ago
  I don't necessarily think it would mean more scientists, but it would mean more expense. You have a moderate number of low impact papers that people are doing for tenure today - papers for the purpose of cranking out papers. We are talking about redirecting efforts but increasing quality of what you have.
mattkrause 3y ago
Longer, even! Some experiments that study biological development or trained animals can take a year or more of fairly intense effort to start generating data.
- Maxion 3y ago
  A year? some data sets take decades to build up before significant papers can be published on their data. Replication of the dataset is just not feasible. This whole thread just shows how little the average HNer knows about the academic sciences.
- tnecniv 3y ago
  I know people that had to take a 6+ month trip to Antarctica for part of their work and others that had to share time on a piece of experimental equipment with a whole department — they got a few weeks per year to run their experiment and had to milk that for all it’s worth. Even if they had funding, that machine required large amounts of space and staff to keep it running and they aren’t off the shelf products — only a few exist at large research centers.
ebiester 3y ago
It's simple but not easy: You create another path to tenure which is based on replication, or on equal terms as a part of a tenure package. (For example, x fewer papers but x number of replications, and you are expected to have x replications in your specialty.) You also create a grant funding section for replication which is then passed on to these independent systems. (You would have to have some sort of randomization handled as well.) Replication has to be considered at the same value as original research. And maybe smaller faculties at R2s pivot to replication hubs. And maybe this is easier for some sections of biology, chemistry and psychology than it is for particle physics. We could start where cost of replication is relatively low and work out the details. It's completely doable in some cases. (It may never be doable in some areas either.)
- Eddy_Viscosity2 3y ago
  It's not easy because it isn't simple. How do get all of the universities to change their incentives to back this?
  - ebiester 3y ago
    We agree - the "simple not easy" turn of phrase is speaking to that point. It is easy once implemented, but it isn't easy to transition. (I am academia-adjacent by marriage but closer to the humanities, so I understand the amount of work it would take to perform the transition.)
    - MichaelZuo 3y ago
      This isn't just not easy, it would probably be extremely political to change the structure of the NSF, National Labs, all universities and colleges, etc., so dramatically.
- SkyMarshal 3y ago
  > x fewer papers but x number of replications, and you are expected to have x replications in your specialty. Could it be simplified it even further to say x number of papers, but they only count if they’re replicated by others in the field?
dongping 3y ago
While it is a lot of work, I tend to think that one can then always publish preprints if they can't wait for the replication. I don't understand why a published paper should count as an achievement (against tenure or funding) at all before the work is replicated. The current model just creates perverse incentives to encourage lying, P-hacking, and cherry-picking. This would at least work for fields like machine learning. This is, of course, a naive proposal without too much thought into it. But I was wondering what I would have missed here.
- i_no_can_eat 3y ago
  and in this proposal, who will be tasked with replicating the work?
  - dongping 3y ago
    In some fields, replication is already the prerequisite to benchmark the SoTA. So the incentives boil down to publishing them along with negative results. Or as some have suggested, make it mandatory for PHD candidates to replicate. Though, it seems that it is possible to game the system, by creating positive/negative replication intentionally, to collude with/harm the author.
kergonath 3y ago
> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. They also tend to over-estimate the effect of peer review (often equating peer review with validity). > If someone cares enough about the work to build on it, they will replicate it anyway. And in that case they have a good incentive to spend the effort. If that works this will indirectly support the original paper even if the following papers don't specifically replicate the original results. Though this part is much more problematic if the following experiments fail, then this will likely remain entirely unpublished. It can also remain unpublished if other things did not work out, even if the results could be replicated. A half-fictional example: a team is working on a revolutionary new material to solve complicated engineering problems. They found a material that was synthesised by someone in the 1980s, published once and never reproduced, which they think could have the specific property they are after. So they synthesise it, and it turns out that the material exists, with the expected structure but not with the property they hoped. They aren’t going to write it up and publish it; they’re just going to scrap it and move on to the next candidate. Different teams might be doing the same thing at the same time, and nobody coming after them will have a clue.
- techdragon 3y ago
  This waste of effort by way of duplicating unpublished negative results is a big factor in why replicated results deserve to be rated more highly than results that have not been replicated regardless of the prestige of the researchers or the institutions involved… if no one can prove your work work was correct… how much can anyone trust your work… I have gone down the rabbit hole of engineering research before and 90% of the time I’ve managed to find an anecdote or subsequent research footnotes or actual subsequent research publications, that substantially invalidated the lofty claims of the engineers in the 70s or 80s (which is amazing still despite this, a genuine treasure trove of research unused and sometimes useful aerospace engineering research and development) and unfortunately outside the few proper publications, a lot of the invalidations are not properly reverse cited research material and I could have spent a week cross referencing before I spot the link and realise the unnamed work they are saying they are proving wrong is actually some footnotes containing the only published data (before their new paper) on some old work that has a bad scan copy on the NASA NTRS server under some obscure title and no related keywords to the topic the research is notionally about… Academic research can genuinely suck sometimes… particularly when you want to actually apply it.
kshahkshah 3y ago
When I looked into this, more than 15 years ago, I thought the difficult portion wasn't sharing the recipe, but the ingredients, if you will - granted I was in a molecular biology lab. Effectively the Material Transfer Agreements between Universities all trying to protect their IP made working with each other unbelievably inefficient. You'd have no idea if you were going down a well trodden path which would yield no success because you have no idea it was well trod. No one publishes negative results, etc.
justinpombrio 3y ago
> If someone cares enough about the work to build on it, they will replicate it anyway. Well, the trouble is that hasn't been the case in practice. A lot of the replication crisis was attempting for the first time to replicate a foundational paper that dozens of other papers took as true and built on top of, and then seeing said foundational paper fail to replicate. The incentives point toward doing new research instead of replication, and that needs to change.
- p1esk 3y ago
  It is the case in my field (ML): if I care enough about a published result I try to replicate it.
  - tnecniv 3y ago
    This is something very sensible in ML since, you likely want to use that algorithm for something else (or to extend / modify it), so you need to get it working in your pipeline and verify it works by comparing with the published result. In something like psychology that is likely harder, since the experiment you want to do might be related to but differ significantly from the prior work. I am no psychologist, but I’d like to think that they don’t take one study as ground truth for that reason but try to understand causal mechanisms with multiple studies as data points. If the hypothesis is correct, it will likely present in multiple ways.
brightball 3y ago
> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. The alternative is a bunch of stuff being published which people belief as "science" that doesn't hold up under scrutiny, which undermines the reliability of science itself. The current approach simply gives people reason to be skeptical.
- ImPostingOnHN 3y ago
  I'm not convinced this proposed alternative is better than the status quo. It's simply not feasible, no matter how many benefits one might imagine. the concern about skepticism is not irrelevant, but many of these skeptics also are skeptical of the earth being round, or older than a few thousand years, or not created by an omnipotent skylord, and I'm not sure it's actually a significant concern given the current number and expertise of those who are skeptical so, we can hear their arguments for their skepticism, but that doesn't mean the arguments are valid to warrant the skepticism exhibited. And in the end, that's what matters: skepticism warranted by valid arguments, not just any Cletus McCletus's skepticism of heliocentrism, as if his opinion is equal to that of an astrophysicist (it isn't). And you know what? It isn't necessary to convince a ditch digger that the earth goes around the sun, if they feel like arguing about it.
  - brightball 3y ago
    > skeptical of the earth being round I’m skeptical that these people truly exist outside of the internet wanting it to be true.
    - ImPostingOnHN 3y ago
      > I’m skeptical that these people truly exist outside of the internet wanting it to be true I'm not, just like I'm not skeptical that climate change denialists truly exist outside of the internet there's simply no valid argument to warrant skepticism of either, given the ease of locating evidence that both do
jofer 3y ago
Also, don't forget that a lot of replication would fundamentally involve going and collecting additional samples / observations / etc in the field area, which is often expensive, time consuming, and logistically difficult. It's not just "can we replicate the analysis on sample X", but also "can we collect a sample similar to X and do we observe similar things in the vicinity" in many cases. That alone may require multiple seasons of rather expensive fieldwork. Then you have tens to hundreds of thousands of dollars in instrument time to pay to run various analysis which are needed in parallel with the field observations. It's rarely the simple data analysis that's flawed and far more frequently subtle issues with everything else. In most cases, rather than try to replicate, it's best to test something slightly different to build confidence in a given hypothesis about what's going on overall. That merits a separate paper and also serves a similar purpose. E.g. don't test "can we observe the same thing at the same place?", and instead test "can we observe something similar/analogous at a different place / under different conditions?". That's the basis of a lot of replication work in geosciences. It's not considered replication, as it's a completely independent body of work, but it serves a similar purpose (and unlike replication studies, it's actually publishable).
deleted 3y ago
[deleted]
DoctorOetker 3y ago
> [...] non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper Either "peer reviewed" articles describe progress of promising results, or they don't. If they don't the research is effectively ignored (at least until someone finds it promising). So let's consider specifically output that described promising results. After "peer review" any apparently promising results prompt other groups to build on them by utilizing it as a step or building block. It can take many failed attempts by independent groups before anyone dares publish the absence of the proclaimed observations, since they may try it over multiple times thinking they must have botched it somewhere. On paper it sounds more expensive to require independent replication, but only because the costs of replication attempts are hidden until its typically rather late. Is it really more expensive if the replication attempts are in some sense mandatory? Or is it perhaps more expensive to pretend science has found a one-shot "peer reviewed" method, resulting in uncoordinated independent reproduction attempts that may go unannounced before, or even after failed replications? The pseudo-final word, end of line? What about the "in some sense mandatory" replication? Perhaps roll provable dice for each article, and in-domain sortition to randomly assign replicators. So every scientist would be spending a certain fraction of their time replicating the research of others. The types of acceptable excuses to derelict these duties should be scrutinized and controlled. But some excuses should be very valid, for example conscientious objection. If you are tasked to reproduce some of Dr. Mengele's works, you can cop out on condition that you thoroughly motivate your ethical concerns and objections. This could also bring a lot of healthy criticism to a lot of practices, which is otherwise just ignored an glossed over for fear of future career opportunities.
omgwtfbyobbq 3y ago
What about a system where peer replication is required if the number of citations exceeds some threshold?
- p1esk 3y ago
  Who will be replicating it? Why would I want to set aside my own research to replicate some claim someone made? How would this help my career?
  - deleted 3y ago
    [deleted]
  - Knee_Pain 3y ago
    Academia's values are not objective. Why is it that replicating or refuting a study is not seen on par as being a co-author of said study? There is nothing set in stone preventing this, only the current academic culture.
    - p1esk 3y ago
      Because I want to do original research, and be known for doing original research. Only if I fail at that, I might settle for being a guy who reproduces others’ work (which basically means the transition from a researcher to an engineer).
      - omgwtfbyobbq 3y ago
        Whether or not you would be doing original research depends on whether the cited work can be replicated. If the cited work is unable to be replicated, and you try to replicate but get different results, then you would be doing original research, and then you can base further work on your initial original study that came to a different result. On the flip side, if you are able to replicate it, then you are doing extra work initially, but after replicating the work you've cited, the work you've done is more likely to be reproducible by someone else. The amount of citations needed to require replication could itself be a function of how easy it is to replicate work across an entire field. A field where there's a high rate of success in replicating work could have a higher threshold for requiring replication compared to a field where it's difficult to replicate work.
johnnyworker 3y ago
> If someone cares enough about the work to build on it, they will replicate it anyway. Does it really deserve to be called work if it doesn't include the a full, working set of instructions that if followed to a T allow it to be replicated? To me that's more like pollution, making it someone else's problem. I certainly don't see how "we did this, just trust us" can even be considered science, and that's not because I don't understand the scientific method, that's because I don't make a living with it, and have no incentive to not rock the boat.
- davidktr 3y ago
  You just described the majority of scientific papers. A "working set of instructions" is not really feasible in most cases. You can't include every piece of hard- and software required to replicate your own setup.
  - johngladtj 3y ago
    You should.
  - lliamander 3y ago
    Sounds like a problem worth solving.
  - johnnyworker 3y ago
    Then don't call it science, since it doesn't contribute anything to the body of human knowledge. I think it's fascinating that we can at the same time hold things like "one is none" to be true, or that you should write tests first, but with science we already got so used to a lack of discipline that we just declare it fine. It's not hard to not climb a tower you can't get down from. It's the default, actually. You start with something small where you can describe everything that goes into replicating it. Then you replicate it yourself, based on your own instructions. Before that, you don't bother anyone else with it. Once that is done, and others can replicate as well, it "actually exists". And if that means the majority of stuff has to be thrown out, I'd suggest doing that sooner rather than later, instead of just accumulating scientific debt.
sebzim4500 3y ago
>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. I think it would be fine to half the productivity of these fields, if it means that you can reasonably expect papers to be accurate.
- hoosieree 3y ago
  Half is wildly optimistic.
- advisedwang 3y ago
  It would be more than just half productivity. Not only do you have to do the work twice, but you add the delay of someone else replicating before something can be published and built upon by others. If you are developer, imagine how much your productivity would drop going from a 3 minute build to a 1 day build.
  - orangepurple 3y ago
    Terrible analogy. It might take months to come up with an idea but another should be able to follow your method and implement it much more quickly than it took you to come up with the concept and implement it.
    - magimas 3y ago
      horrible take. Taking the LK99 situation as an example: simply copying and adapting a well described growth recipee to your own setup and lab conditions may take weeks. And how would you address situations where measurement setups only exist once on the earth? How would you do peer replication of LHC measurements? Wait for 50 years till the next super-collider is built and someone else can finally verify the results? On a smaller scale: If you need measurements at a synchrotron radiation source to replicate a measurement, is someone supposed to give up his precious measurement time to replicate a paper he isn't interested in? And is the original author of a paper that's in the queue for peer replication supposed to wait for a year or two till the reviewer gets a beamtime on an appropriate measurement station? Even smaller: I did my PhD in a lab with a specific setup that only a single other group in the world had an equivalent to. You simply would not be able to replicate these results. Peer replication is completely unfeasible in experimental fields of science. The current process of peer review is alright, people just need to learn that single papers standing by themselves don't mean too much. The "peer replication" happens over time anyway when others use the same tools, samples, techniques on related problems and find results in agreement with earlier papers.
coding123 3y ago
Maybe doing an experiment twice, even with a cost that is double, makes more sense so that we don't all throw away our coffee when coffee is bad, or throw away our gluten when gluten is bad, etc... (those are trivial examples) basically the cost to perform the science in many cases is so minuscule in scale to how it could affect society.
- pvaldes 3y ago
  One. Doing experiments is yet enough difficult and painful. Two. This drain of resources can't be done for free. Somebody will need to pay twice for half of the research [1], and faster. Peers will need to be hired and paid, maybe by the writer's grants. Researchers cant justify to give their own funds to other teams without a profound change in regulation and even in that case would be harming their own projects. [1] as the valuable experts are now stuck validating things instead doing their own job Would open also a door for foul play. Blocking competitors teams in molasses just trowing them secondary silly problems that they know that are a dead end, while the other team work in the real deal, and take the advantage to win the patent.
boxed 3y ago
> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. I don't see how the current system works really either. Fraud is rampant, and replication crisis is the most common state of most fields. Basically the current system is failing at finding out what is true. Which is the entire point. That's pretty damn bad.
- tptacek 3y ago
  Fraud seems rampant because you hear about cases of fraud, but not about the tens of thousands of research labs plugging away day after day.
  - lliamander 3y ago
    I agree that most labs are probably not out to defraud people. But without replication I don't think it's reasonable to have much confidence in what is published.
    - tptacek 3y ago
      That's not what publication is about. Publication is a conversation with other researchers; it is part of the process of reaching the truth, not its endpoint.
      - cpach 3y ago
        People in general (at least on da Internetz) seem to focus way to much on single studies, and way too little on meta-studies. AFAICT meta-studies is the level where we as a society really can try to say something intelligent about how stuff works. If an important question is not included in a meta-study, we (i.e. universities and research labs) probably need to do more research on that topic before we really can say that much about it.
      - lliamander 3y ago
        Sure, and scientists need a place to have such conversations. But publication is not a closed system. The "published, peer-reviewed paper" is frequently an artifact used to decide practical policy matters in many institutions both public and private. To the extent that Science (as an institution in its own right) wants to influence policy, that influence needs to be grounded in reproducible results. Also, I would not be surprised if stronger emphasis on reproducibility improved the quality of conversation among scientists.
b59831 3y ago
[dead]
sqrt_1 3y ago
FYI there is a at least one science journal that only publishes reproduced research: Organic Syntheses "A unique feature of the review process is that all of the data and experiments reported in an article must be successfully repeated in the laboratory of a member of the editorial board as a check for reproducibility prior to publication" https://en.wikipedia.org/wiki/Organic_Syntheses https://en.wikipedia.org/wiki/Organic_Syntheses
- jamesash 3y ago
  Started in 1924 and still going strong 100 years later. The gold standard for organic chemistry procedures. "If you can't reproduce a procedure in Org Syn, it's YOUR fault" - my PhD supervisor
RugnirViking 3y ago
lets be brutally honest with ourselves. 99% of all papers mean nothing. They add nothing to the collective knowledge of humanity. In my field of robotics there are SOOO many papers that are basically taking three or four established algorithms/machine learning models, and applying them to off-the-shelf hardware. The kind of thing any person educated in the field could almost guess the results exactly. Hundreds of such iterations for any reasonably popular problems space (prosthetics, drones for wildfires, museum guide robot) etc every month. Far more than could possibly be useful to anyone. There should probably be some sort of separate process for things that actually claim to make important discoveries. I don't know what or how that should work. In all honesty maybe there should just be less papers, however that could be achieved.
- indymike 3y ago
  > 99% of all papers mean nothing. They add nothing to the collective knowledge of humanity. A lot of papers are done as a part of the process of getting a degree or keeping or getting job. The value is mostly the candidate showing they have the acumen to produce a paper of such quality that meets the publisher and peer review requirements. In some cases, it is to show a future employer some level of accomplishment or renown. The knowledge for humanity is mostly the authors ability to get published.
  - RugnirViking 3y ago
    well yes. But these should go somewhere else than the papers that may actually contain significant results. The problem we have here is that there is an enormous quantity of such useless papers mixed in with the ones actually trying to do science. I understand that part of the reason for that is that people need to appear as though they are part of the "actually trying" crowd to get the desired job effects. But it is nonetheless a problem, and a large one very worth at least trying to solve.
    - indymike 3y ago
      > I understand that part of the reason for that is that people need to appear as though they are part of the "actually trying" crowd to get the desired job effects There's another less obvious problem: we don't always know what is groundbreaking and what is not until something is published and is accepted at large as being a really big deal.
throwaway4aday 3y ago
What's the value in publishing something that is never replicated? If no one ever reproduces the experiment and gets the same results then you don't know if any interpretations based on that experiment are valid. It would also mean that whatever practical applications could have come from the experiment are never realized. It makes the entire pursuit seem completely useless.
- wizofaus 3y ago
  > What's the value in publishing something that is never replicated? Because it presents an experimental result to other scientists that they may consider worth trying to replicate?
  - dongping 3y ago
    Then those unconfirmed results are better put on arxiv, instead of being used to evaluate the performance of scientists. Tenure and grant committees should only consider replicated work.
    - geysersam 3y ago
      I don't agree. A published article should not be taken for Gods Truth no matter if it's replicated or peer reviewed. Lots of "replicated" "peer-reviewed" research have been found to be wrong. That's fine, it's part of the process of discovery. A paper should be taken for what it is: a piece of scientific work, a part of a puzzle.
- geysersam 3y ago
  It still has value if we assume the experiment was done by competent honest people who are unlikely to try to fool us on purpose and unlikely do have made errors. It would be even better if it was replicated of course. Depending on what certainty you need you might have to wait for the result of one or several replications, but that is application dependent.
indymike 3y ago
> This is also all work that doesn't benefit the scientists replicating the paper. It only costs them money and time. Maybe this is what needs to change. If we only reward discovery and success, then the incentive is to only produce discovery and success.
techas 3y ago
Well, you could put incentives to make replication attractive. Give credit for replication. Give money to the researchers doing the replication/review. Today we pay an average of 2000€ per article, reviewers get 0€ and the editorial keeps all for putting a pdf online. I would say there is margin there to invest in improving the review process.
- mandmandam 3y ago
  It's wild to me that although we know that it was Ghislaine Maxwell's daddy who started this incredibly corrupt system, people hardly mention this fact. The US system, and others, even attack people who dare to try and make science more open. RIP Aaron Swartz, and long live Alexandra Elbakyan.
throwawaymaths 3y ago
> I don't see how this could ever work, http://www.orgsyn.org/ http://www.orgsyn.org/ > All procedures and characterization data in OrgSyn are peer-reviewed and checked for reproducibility in the laboratory of a member of the Board of Editors Never is a strong word.
majormajor 3y ago
I think the current system is just measuring entirely the wrong thing. Yes, fewer papers would be published. But today's goal is "publish papers" not "learn and disseminate truly useful and novel things", and while this doesn't solve it entirely, it pushes incentives further away from "publish whatever pure crap you can get away with." You get what you measure -> sometimes you need to change what/how you measure. > If someone cares enough about the work to build on it, they will replicate it anyway. That's duplicative at the "oh maybe this will be useful to me" stage, with N different people trying to replicate. And with replication not a first-class part of the system, the effort of replication (e_R) is high. For appealing things, N is probably > 2. So N X e_R total effort. If you move the burden at the "replicate to publish" stage, you can fix the number of replicas needed so N=2 (or whatever) and you incentive the orginal researchers to make e_R lower (which will improve the quality of their research even before the submit-for-publication stage). I've been in the system, I spent a year or two chasing the tail of rewrites, submissions, etc, for something that was detectable as low-effect-size in the first place but I was told would still be publishable. I found out as part of that that it would only sometimes yield a good p-value! And everything in the system incentivized me to hide that for as long as possible, instead of incentivizing me to look for something else or make it easy for others to replicate and judge for themselves. Hell, do something like "give undergrads the opportunity to earn Master's on top of their BSes, say, by replicating (or blowing holes in) other people's submissions." I would've eaten up an opportunity like that to go really really deep* in some specialized area in exchange for a masters degree in a less-structured way than "just take a bunch more courses."
jononomo 3y ago
If it is not replicated it shouldn't be published, other than as a provisional draft. I don't care if it hurts your feelings.
iamthemonster 3y ago
My Master's thesis was basically taking a purely theoretical paper and "replicating" it, by which I mean taking the formulae and just writing the software to run them. It sounds trivial to an outsider but even that was I guess 300 hours of work. In general I think undergraduate projects are a great space to attempt to replicate findings, but it heavily depends on the field. Fundamental physics experiments can be expensive and require equipment that's outside the reach of undergrads. But one thing I love about engineering as an academic field, by comparison, is that anything you research tends to be more achievable for others to replicate because as your end goal you are aiming for something that's practical in the field.
coldtea 3y ago
>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. Then perhaps those papers shouldn't be published? Or held in any higher esteem than a blog post by the same authors?
- gus_massa 3y ago
  An arXiv preprint is like a blog post. A paper in a peer review journal is like posting a request for reproduction in a heavily moderated mailing list. A paper in a predatory journal is like the "You are the best ___" price that you get if you pay to go to the "congress" invitation in spam. Neither of them guaranty that the result is true. The publication in some peer review journals give a minimal guaranty that the paper is not horribly bad, but I've seen too much crap there too. I know a few journals and author in my area that are serious and I can guess the result will hold, but I find very difficult to evaluate journals and authors in other areas.
wilde 3y ago
The pressure to replicate would make folks publish things in forms that are easier to replicate. This cost would go down over time.
faeriechangling 3y ago
In some fields research can’t be replicated later. Much of all autism research will NEVER be replicated because the population of those considered autistic is not stable over time. Other research proves impossible to replicate because whatever experiment was not described in enough detail to actually replicate it, which should be grounds to immediately dismiss the research before publishing, but which can’t truly be caught if you don’t actually try to reproduce. Finally these practical concerns don’t even touch on the biggest benefit of reproduction as standard which is that almost nobody wants to reproduce research as they are not rewarded for doing so. This would give somebody, namely those who want to publish something, a strong impetus to get that reproduction done which wouldn’t otherwise exist.
chmod600 3y ago
Please excuse my ignorance, but I'm not convinced. What are we supposed to do in a hundred years when the scientists of today are dead and we have a bunch of results with important implications that aren't documented well enough to replicate?
ljf 3y ago
Indeed, a friend of my was running experiments on quantum computing - the first 18 months on each run of his tests, as he moved between jobs and universities, was just setting up the necessary refrigeration systems (and generally building them from scratch each time). Only then could he even start building the experiment - total time to run it all seems to run across years.
picadores 3y ago
Yes, the amount of work done that could go into more paper churn?
oldgradstudent 3y ago
> This is also all work that doesn't benefit the scientists replicating the paper. It only costs them money and time If you build upon a result, you almost have to replicate it. An acquaintance spent years building upon a result that turned out to be fraudulent/p-hacked.

backtoyoujim 3y ago

Yes it would indeed mean slowing down and having more scientists. It would mean disruption is no longer a useful tool for human development.

brnaftr361 3y ago
It may not be. I would be willing to argue that there was a tipping point and we've long exceeded its boundary - progress and disruption now is just making finding an equilibrium in the future increasingly difficult. So entering into a paradigm where we test the known space - especially presently - would 1) help reduce cruft; 2) abate undersirable forward progress; 3) train the next generation(s) of scientists to be more diligent and better custodians of the domain.
ebiester 3y ago
I don't necessarily think it would mean more scientists, but it would mean more expense. You have a moderate number of low impact papers that people are doing for tenure today - papers for the purpose of cranking out papers. We are talking about redirecting efforts but increasing quality of what you have.

mattkrause 3y ago

Longer, even! Some experiments that study biological development or trained animals can take a year or more of fairly intense effort to start generating data.

Maxion 3y ago
A year? some data sets take decades to build up before significant papers can be published on their data. Replication of the dataset is just not feasible. This whole thread just shows how little the average HNer knows about the academic sciences.
tnecniv 3y ago
I know people that had to take a 6+ month trip to Antarctica for part of their work and others that had to share time on a piece of experimental equipment with a whole department — they got a few weeks per year to run their experiment and had to milk that for all it’s worth. Even if they had funding, that machine required large amounts of space and staff to keep it running and they aren’t off the shelf products — only a few exist at large research centers.

ebiester 3y ago

It's simple but not easy: You create another path to tenure which is based on replication, or on equal terms as a part of a tenure package. (For example, x fewer papers but x number of replications, and you are expected to have x replications in your specialty.) You also create a grant funding section for replication which is then passed on to these independent systems. (You would have to have some sort of randomization handled as well.) Replication has to be considered at the same value as original research. And maybe smaller faculties at R2s pivot to replication hubs. And maybe this is easier for some sections of biology, chemistry and psychology than it is for particle physics. We could start where cost of replication is relatively low and work out the details. It's completely doable in some cases. (It may never be doable in some areas either.)

Eddy_Viscosity2 3y ago
It's not easy because it isn't simple. How do get all of the universities to change their incentives to back this?
- ebiester 3y ago
  We agree - the "simple not easy" turn of phrase is speaking to that point. It is easy once implemented, but it isn't easy to transition. (I am academia-adjacent by marriage but closer to the humanities, so I understand the amount of work it would take to perform the transition.)
  - MichaelZuo 3y ago
    This isn't just not easy, it would probably be extremely political to change the structure of the NSF, National Labs, all universities and colleges, etc., so dramatically.
SkyMarshal 3y ago
> x fewer papers but x number of replications, and you are expected to have x replications in your specialty. Could it be simplified it even further to say x number of papers, but they only count if they’re replicated by others in the field?

dongping 3y ago

While it is a lot of work, I tend to think that one can then always publish preprints if they can't wait for the replication. I don't understand why a published paper should count as an achievement (against tenure or funding) at all before the work is replicated. The current model just creates perverse incentives to encourage lying, P-hacking, and cherry-picking. This would at least work for fields like machine learning. This is, of course, a naive proposal without too much thought into it. But I was wondering what I would have missed here.

i_no_can_eat 3y ago
and in this proposal, who will be tasked with replicating the work?
- dongping 3y ago
  In some fields, replication is already the prerequisite to benchmark the SoTA. So the incentives boil down to publishing them along with negative results. Or as some have suggested, make it mandatory for PHD candidates to replicate. Though, it seems that it is possible to game the system, by creating positive/negative replication intentionally, to collude with/harm the author.

kergonath 3y ago

> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. They also tend to over-estimate the effect of peer review (often equating peer review with validity). > If someone cares enough about the work to build on it, they will replicate it anyway. And in that case they have a good incentive to spend the effort. If that works this will indirectly support the original paper even if the following papers don't specifically replicate the original results. Though this part is much more problematic if the following experiments fail, then this will likely remain entirely unpublished. It can also remain unpublished if other things did not work out, even if the results could be replicated. A half-fictional example: a team is working on a revolutionary new material to solve complicated engineering problems. They found a material that was synthesised by someone in the 1980s, published once and never reproduced, which they think could have the specific property they are after. So they synthesise it, and it turns out that the material exists, with the expected structure but not with the property they hoped. They aren’t going to write it up and publish it; they’re just going to scrap it and move on to the next candidate. Different teams might be doing the same thing at the same time, and nobody coming after them will have a clue.

techdragon 3y ago
This waste of effort by way of duplicating unpublished negative results is a big factor in why replicated results deserve to be rated more highly than results that have not been replicated regardless of the prestige of the researchers or the institutions involved… if no one can prove your work work was correct… how much can anyone trust your work… I have gone down the rabbit hole of engineering research before and 90% of the time I’ve managed to find an anecdote or subsequent research footnotes or actual subsequent research publications, that substantially invalidated the lofty claims of the engineers in the 70s or 80s (which is amazing still despite this, a genuine treasure trove of research unused and sometimes useful aerospace engineering research and development) and unfortunately outside the few proper publications, a lot of the invalidations are not properly reverse cited research material and I could have spent a week cross referencing before I spot the link and realise the unnamed work they are saying they are proving wrong is actually some footnotes containing the only published data (before their new paper) on some old work that has a bad scan copy on the NASA NTRS server under some obscure title and no related keywords to the topic the research is notionally about… Academic research can genuinely suck sometimes… particularly when you want to actually apply it.

kshahkshah 3y ago

When I looked into this, more than 15 years ago, I thought the difficult portion wasn't sharing the recipe, but the ingredients, if you will - granted I was in a molecular biology lab. Effectively the Material Transfer Agreements between Universities all trying to protect their IP made working with each other unbelievably inefficient. You'd have no idea if you were going down a well trodden path which would yield no success because you have no idea it was well trod. No one publishes negative results, etc.

justinpombrio 3y ago

> If someone cares enough about the work to build on it, they will replicate it anyway. Well, the trouble is that hasn't been the case in practice. A lot of the replication crisis was attempting for the first time to replicate a foundational paper that dozens of other papers took as true and built on top of, and then seeing said foundational paper fail to replicate. The incentives point toward doing new research instead of replication, and that needs to change.

p1esk 3y ago
It is the case in my field (ML): if I care enough about a published result I try to replicate it.
- tnecniv 3y ago
  This is something very sensible in ML since, you likely want to use that algorithm for something else (or to extend / modify it), so you need to get it working in your pipeline and verify it works by comparing with the published result. In something like psychology that is likely harder, since the experiment you want to do might be related to but differ significantly from the prior work. I am no psychologist, but I’d like to think that they don’t take one study as ground truth for that reason but try to understand causal mechanisms with multiple studies as data points. If the hypothesis is correct, it will likely present in multiple ways.

brightball 3y ago

> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. The alternative is a bunch of stuff being published which people belief as "science" that doesn't hold up under scrutiny, which undermines the reliability of science itself. The current approach simply gives people reason to be skeptical.

ImPostingOnHN 3y ago
I'm not convinced this proposed alternative is better than the status quo. It's simply not feasible, no matter how many benefits one might imagine. the concern about skepticism is not irrelevant, but many of these skeptics also are skeptical of the earth being round, or older than a few thousand years, or not created by an omnipotent skylord, and I'm not sure it's actually a significant concern given the current number and expertise of those who are skeptical so, we can hear their arguments for their skepticism, but that doesn't mean the arguments are valid to warrant the skepticism exhibited. And in the end, that's what matters: skepticism warranted by valid arguments, not just any Cletus McCletus's skepticism of heliocentrism, as if his opinion is equal to that of an astrophysicist (it isn't). And you know what? It isn't necessary to convince a ditch digger that the earth goes around the sun, if they feel like arguing about it.
- brightball 3y ago
  > skeptical of the earth being round I’m skeptical that these people truly exist outside of the internet wanting it to be true.
  - ImPostingOnHN 3y ago
    > I’m skeptical that these people truly exist outside of the internet wanting it to be true I'm not, just like I'm not skeptical that climate change denialists truly exist outside of the internet there's simply no valid argument to warrant skepticism of either, given the ease of locating evidence that both do

jofer 3y ago

Also, don't forget that a lot of replication would fundamentally involve going and collecting additional samples / observations / etc in the field area, which is often expensive, time consuming, and logistically difficult. It's not just "can we replicate the analysis on sample X", but also "can we collect a sample similar to X and do we observe similar things in the vicinity" in many cases. That alone may require multiple seasons of rather expensive fieldwork. Then you have tens to hundreds of thousands of dollars in instrument time to pay to run various analysis which are needed in parallel with the field observations. It's rarely the simple data analysis that's flawed and far more frequently subtle issues with everything else. In most cases, rather than try to replicate, it's best to test something slightly different to build confidence in a given hypothesis about what's going on overall. That merits a separate paper and also serves a similar purpose. E.g. don't test "can we observe the same thing at the same place?", and instead test "can we observe something similar/analogous at a different place / under different conditions?". That's the basis of a lot of replication work in geosciences. It's not considered replication, as it's a completely independent body of work, but it serves a similar purpose (and unlike replication studies, it's actually publishable).

deleted 3y ago

[deleted]

DoctorOetker 3y ago

> [...] non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper Either "peer reviewed" articles describe progress of promising results, or they don't. If they don't the research is effectively ignored (at least until someone finds it promising). So let's consider specifically output that described promising results. After "peer review" any apparently promising results prompt other groups to build on them by utilizing it as a step or building block. It can take many failed attempts by independent groups before anyone dares publish the absence of the proclaimed observations, since they may try it over multiple times thinking they must have botched it somewhere. On paper it sounds more expensive to require independent replication, but only because the costs of replication attempts are hidden until its typically rather late. Is it really more expensive if the replication attempts are in some sense mandatory? Or is it perhaps more expensive to pretend science has found a one-shot "peer reviewed" method, resulting in uncoordinated independent reproduction attempts that may go unannounced before, or even after failed replications? The pseudo-final word, end of line? What about the "in some sense mandatory" replication? Perhaps roll provable dice for each article, and in-domain sortition to randomly assign replicators. So every scientist would be spending a certain fraction of their time replicating the research of others. The types of acceptable excuses to derelict these duties should be scrutinized and controlled. But some excuses should be very valid, for example conscientious objection. If you are tasked to reproduce some of Dr. Mengele's works, you can cop out on condition that you thoroughly motivate your ethical concerns and objections. This could also bring a lot of healthy criticism to a lot of practices, which is otherwise just ignored an glossed over for fear of future career opportunities.

omgwtfbyobbq 3y ago

What about a system where peer replication is required if the number of citations exceeds some threshold?

p1esk 3y ago
Who will be replicating it? Why would I want to set aside my own research to replicate some claim someone made? How would this help my career?
- deleted 3y ago
  [deleted]
- Knee_Pain 3y ago
  Academia's values are not objective. Why is it that replicating or refuting a study is not seen on par as being a co-author of said study? There is nothing set in stone preventing this, only the current academic culture.
  - p1esk 3y ago
    Because I want to do original research, and be known for doing original research. Only if I fail at that, I might settle for being a guy who reproduces others’ work (which basically means the transition from a researcher to an engineer).
    - omgwtfbyobbq 3y ago
      Whether or not you would be doing original research depends on whether the cited work can be replicated. If the cited work is unable to be replicated, and you try to replicate but get different results, then you would be doing original research, and then you can base further work on your initial original study that came to a different result. On the flip side, if you are able to replicate it, then you are doing extra work initially, but after replicating the work you've cited, the work you've done is more likely to be reproducible by someone else. The amount of citations needed to require replication could itself be a function of how easy it is to replicate work across an entire field. A field where there's a high rate of success in replicating work could have a higher threshold for requiring replication compared to a field where it's difficult to replicate work.

johnnyworker 3y ago

> If someone cares enough about the work to build on it, they will replicate it anyway. Does it really deserve to be called work if it doesn't include the a full, working set of instructions that if followed to a T allow it to be replicated? To me that's more like pollution, making it someone else's problem. I certainly don't see how "we did this, just trust us" can even be considered science, and that's not because I don't understand the scientific method, that's because I don't make a living with it, and have no incentive to not rock the boat.

davidktr 3y ago
You just described the majority of scientific papers. A "working set of instructions" is not really feasible in most cases. You can't include every piece of hard- and software required to replicate your own setup.
- johngladtj 3y ago
  You should.
- lliamander 3y ago
  Sounds like a problem worth solving.
- johnnyworker 3y ago
  Then don't call it science, since it doesn't contribute anything to the body of human knowledge. I think it's fascinating that we can at the same time hold things like "one is none" to be true, or that you should write tests first, but with science we already got so used to a lack of discipline that we just declare it fine. It's not hard to not climb a tower you can't get down from. It's the default, actually. You start with something small where you can describe everything that goes into replicating it. Then you replicate it yourself, based on your own instructions. Before that, you don't bother anyone else with it. Once that is done, and others can replicate as well, it "actually exists". And if that means the majority of stuff has to be thrown out, I'd suggest doing that sooner rather than later, instead of just accumulating scientific debt.

sebzim4500 3y ago

>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. I think it would be fine to half the productivity of these fields, if it means that you can reasonably expect papers to be accurate.

hoosieree 3y ago
Half is wildly optimistic.
advisedwang 3y ago
It would be more than just half productivity. Not only do you have to do the work twice, but you add the delay of someone else replicating before something can be published and built upon by others. If you are developer, imagine how much your productivity would drop going from a 3 minute build to a 1 day build.
- orangepurple 3y ago
  Terrible analogy. It might take months to come up with an idea but another should be able to follow your method and implement it much more quickly than it took you to come up with the concept and implement it.
  - magimas 3y ago
    horrible take. Taking the LK99 situation as an example: simply copying and adapting a well described growth recipee to your own setup and lab conditions may take weeks. And how would you address situations where measurement setups only exist once on the earth? How would you do peer replication of LHC measurements? Wait for 50 years till the next super-collider is built and someone else can finally verify the results? On a smaller scale: If you need measurements at a synchrotron radiation source to replicate a measurement, is someone supposed to give up his precious measurement time to replicate a paper he isn't interested in? And is the original author of a paper that's in the queue for peer replication supposed to wait for a year or two till the reviewer gets a beamtime on an appropriate measurement station? Even smaller: I did my PhD in a lab with a specific setup that only a single other group in the world had an equivalent to. You simply would not be able to replicate these results. Peer replication is completely unfeasible in experimental fields of science. The current process of peer review is alright, people just need to learn that single papers standing by themselves don't mean too much. The "peer replication" happens over time anyway when others use the same tools, samples, techniques on related problems and find results in agreement with earlier papers.

coding123 3y ago

Maybe doing an experiment twice, even with a cost that is double, makes more sense so that we don't all throw away our coffee when coffee is bad, or throw away our gluten when gluten is bad, etc... (those are trivial examples) basically the cost to perform the science in many cases is so minuscule in scale to how it could affect society.

pvaldes 3y ago
One. Doing experiments is yet enough difficult and painful. Two. This drain of resources can't be done for free. Somebody will need to pay twice for half of the research [1], and faster. Peers will need to be hired and paid, maybe by the writer's grants. Researchers cant justify to give their own funds to other teams without a profound change in regulation and even in that case would be harming their own projects. [1] as the valuable experts are now stuck validating things instead doing their own job Would open also a door for foul play. Blocking competitors teams in molasses just trowing them secondary silly problems that they know that are a dead end, while the other team work in the real deal, and take the advantage to win the patent.

boxed 3y ago

> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. I don't see how the current system works really either. Fraud is rampant, and replication crisis is the most common state of most fields. Basically the current system is failing at finding out what is true. Which is the entire point. That's pretty damn bad.

tptacek 3y ago
Fraud seems rampant because you hear about cases of fraud, but not about the tens of thousands of research labs plugging away day after day.
- lliamander 3y ago
  I agree that most labs are probably not out to defraud people. But without replication I don't think it's reasonable to have much confidence in what is published.
  - tptacek 3y ago
    That's not what publication is about. Publication is a conversation with other researchers; it is part of the process of reaching the truth, not its endpoint.
    - cpach 3y ago
      People in general (at least on da Internetz) seem to focus way to much on single studies, and way too little on meta-studies. AFAICT meta-studies is the level where we as a society really can try to say something intelligent about how stuff works. If an important question is not included in a meta-study, we (i.e. universities and research labs) probably need to do more research on that topic before we really can say that much about it.
    - lliamander 3y ago
      Sure, and scientists need a place to have such conversations. But publication is not a closed system. The "published, peer-reviewed paper" is frequently an artifact used to decide practical policy matters in many institutions both public and private. To the extent that Science (as an institution in its own right) wants to influence policy, that influence needs to be grounded in reproducible results. Also, I would not be surprised if stronger emphasis on reproducibility improved the quality of conversation among scientists.

b59831 3y ago

[dead]

sqrt_1 3y ago

FYI there is a at least one science journal that only publishes reproduced research: Organic Syntheses "A unique feature of the review process is that all of the data and experiments reported in an article must be successfully repeated in the laboratory of a member of the editorial board as a check for reproducibility prior to publication" https://en.wikipedia.org/wiki/Organic_Syntheses https://en.wikipedia.org/wiki/Organic_Syntheses

jamesash 3y ago
Started in 1924 and still going strong 100 years later. The gold standard for organic chemistry procedures. "If you can't reproduce a procedure in Org Syn, it's YOUR fault" - my PhD supervisor

RugnirViking 3y ago

lets be brutally honest with ourselves. 99% of all papers mean nothing. They add nothing to the collective knowledge of humanity. In my field of robotics there are SOOO many papers that are basically taking three or four established algorithms/machine learning models, and applying them to off-the-shelf hardware. The kind of thing any person educated in the field could almost guess the results exactly. Hundreds of such iterations for any reasonably popular problems space (prosthetics, drones for wildfires, museum guide robot) etc every month. Far more than could possibly be useful to anyone. There should probably be some sort of separate process for things that actually claim to make important discoveries. I don't know what or how that should work. In all honesty maybe there should just be less papers, however that could be achieved.

indymike 3y ago
> 99% of all papers mean nothing. They add nothing to the collective knowledge of humanity. A lot of papers are done as a part of the process of getting a degree or keeping or getting job. The value is mostly the candidate showing they have the acumen to produce a paper of such quality that meets the publisher and peer review requirements. In some cases, it is to show a future employer some level of accomplishment or renown. The knowledge for humanity is mostly the authors ability to get published.
- RugnirViking 3y ago
  well yes. But these should go somewhere else than the papers that may actually contain significant results. The problem we have here is that there is an enormous quantity of such useless papers mixed in with the ones actually trying to do science. I understand that part of the reason for that is that people need to appear as though they are part of the "actually trying" crowd to get the desired job effects. But it is nonetheless a problem, and a large one very worth at least trying to solve.
  - indymike 3y ago
    > I understand that part of the reason for that is that people need to appear as though they are part of the "actually trying" crowd to get the desired job effects There's another less obvious problem: we don't always know what is groundbreaking and what is not until something is published and is accepted at large as being a really big deal.

throwaway4aday 3y ago

What's the value in publishing something that is never replicated? If no one ever reproduces the experiment and gets the same results then you don't know if any interpretations based on that experiment are valid. It would also mean that whatever practical applications could have come from the experiment are never realized. It makes the entire pursuit seem completely useless.

wizofaus 3y ago
> What's the value in publishing something that is never replicated? Because it presents an experimental result to other scientists that they may consider worth trying to replicate?
- dongping 3y ago
  Then those unconfirmed results are better put on arxiv, instead of being used to evaluate the performance of scientists. Tenure and grant committees should only consider replicated work.
  - geysersam 3y ago
    I don't agree. A published article should not be taken for Gods Truth no matter if it's replicated or peer reviewed. Lots of "replicated" "peer-reviewed" research have been found to be wrong. That's fine, it's part of the process of discovery. A paper should be taken for what it is: a piece of scientific work, a part of a puzzle.
geysersam 3y ago
It still has value if we assume the experiment was done by competent honest people who are unlikely to try to fool us on purpose and unlikely do have made errors. It would be even better if it was replicated of course. Depending on what certainty you need you might have to wait for the result of one or several replications, but that is application dependent.

indymike 3y ago

> This is also all work that doesn't benefit the scientists replicating the paper. It only costs them money and time. Maybe this is what needs to change. If we only reward discovery and success, then the incentive is to only produce discovery and success.

techas 3y ago

Well, you could put incentives to make replication attractive. Give credit for replication. Give money to the researchers doing the replication/review. Today we pay an average of 2000€ per article, reviewers get 0€ and the editorial keeps all for putting a pdf online. I would say there is margin there to invest in improving the review process.

mandmandam 3y ago
It's wild to me that although we know that it was Ghislaine Maxwell's daddy who started this incredibly corrupt system, people hardly mention this fact. The US system, and others, even attack people who dare to try and make science more open. RIP Aaron Swartz, and long live Alexandra Elbakyan.

throwawaymaths 3y ago

> I don't see how this could ever work, http://www.orgsyn.org/ http://www.orgsyn.org/ > All procedures and characterization data in OrgSyn are peer-reviewed and checked for reproducibility in the laboratory of a member of the Board of Editors Never is a strong word.

majormajor 3y ago

I think the current system is just measuring entirely the wrong thing. Yes, fewer papers would be published. But today's goal is "publish papers" not "learn and disseminate truly useful and novel things", and while this doesn't solve it entirely, it pushes incentives further away from "publish whatever pure crap you can get away with." You get what you measure -> sometimes you need to change what/how you measure. > If someone cares enough about the work to build on it, they will replicate it anyway. That's duplicative at the "oh maybe this will be useful to me" stage, with N different people trying to replicate. And with replication not a first-class part of the system, the effort of replication (e_R) is high. For appealing things, N is probably > 2. So N X e_R total effort. If you move the burden at the "replicate to publish" stage, you can fix the number of replicas needed so N=2 (or whatever) and you incentive the orginal researchers to make e_R lower (which will improve the quality of their research even before the submit-for-publication stage). I've been in the system, I spent a year or two chasing the tail of rewrites, submissions, etc, for something that was detectable as low-effect-size in the first place but I was told would still be publishable. I found out as part of that that it would only sometimes yield a good p-value! And everything in the system incentivized me to hide that for as long as possible, instead of incentivizing me to look for something else or make it easy for others to replicate and judge for themselves. Hell, do something like "give undergrads the opportunity to earn Master's on top of their BSes, say, by replicating (or blowing holes in) other people's submissions." I would've eaten up an opportunity like that to go really really deep* in some specialized area in exchange for a masters degree in a less-structured way than "just take a bunch more courses."

jononomo 3y ago

If it is not replicated it shouldn't be published, other than as a provisional draft. I don't care if it hurts your feelings.

iamthemonster 3y ago

My Master's thesis was basically taking a purely theoretical paper and "replicating" it, by which I mean taking the formulae and just writing the software to run them. It sounds trivial to an outsider but even that was I guess 300 hours of work. In general I think undergraduate projects are a great space to attempt to replicate findings, but it heavily depends on the field. Fundamental physics experiments can be expensive and require equipment that's outside the reach of undergrads. But one thing I love about engineering as an academic field, by comparison, is that anything you research tends to be more achievable for others to replicate because as your end goal you are aiming for something that's practical in the field.

coldtea 3y ago

>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. Then perhaps those papers shouldn't be published? Or held in any higher esteem than a blog post by the same authors?

gus_massa 3y ago
An arXiv preprint is like a blog post. A paper in a peer review journal is like posting a request for reproduction in a heavily moderated mailing list. A paper in a predatory journal is like the "You are the best ___" price that you get if you pay to go to the "congress" invitation in spam. Neither of them guaranty that the result is true. The publication in some peer review journals give a minimal guaranty that the paper is not horribly bad, but I've seen too much crap there too. I know a few journals and author in my area that are serious and I can guess the result will hold, but I find very difficult to evaluate journals and authors in other areas.

wilde 3y ago

The pressure to replicate would make folks publish things in forms that are easier to replicate. This cost would go down over time.

faeriechangling 3y ago

In some fields research can’t be replicated later. Much of all autism research will NEVER be replicated because the population of those considered autistic is not stable over time. Other research proves impossible to replicate because whatever experiment was not described in enough detail to actually replicate it, which should be grounds to immediately dismiss the research before publishing, but which can’t truly be caught if you don’t actually try to reproduce. Finally these practical concerns don’t even touch on the biggest benefit of reproduction as standard which is that almost nobody wants to reproduce research as they are not rewarded for doing so. This would give somebody, namely those who want to publish something, a strong impetus to get that reproduction done which wouldn’t otherwise exist.

chmod600 3y ago

Please excuse my ignorance, but I'm not convinced. What are we supposed to do in a hundred years when the scientists of today are dead and we have a bunch of results with important implications that aren't documented well enough to replicate?

ljf 3y ago

Indeed, a friend of my was running experiments on quantum computing - the first 18 months on each run of his tests, as he moved between jobs and universities, was just setting up the necessary refrigeration systems (and generally building them from scratch each time). Only then could he even start building the experiment - total time to run it all seems to run across years.

picadores 3y ago

Yes, the amount of work done that could go into more paper churn?

oldgradstudent 3y ago

> This is also all work that doesn't benefit the scientists replicating the paper. It only costs them money and time If you build upon a result, you almost have to replicate it. An acquaintance spent years building upon a result that turned out to be fraudulent/p-hacked.