12 ms·
It kinda skips over how large mainstream journals, with their restrictive and often arbitrary standards, have contributed to this. Most will refuse to publish
by RobotToaster 7mo ago
It kinda skips over how large mainstream journals, with their restrictive and often arbitrary standards, have contributed to this. Most will refuse to publish replications, negative studies, or anything they deem unimportant, even if the study was conducted correctly.
- tppiotrowski 7mo agoMaybe we need a journal completely dedicated to replication studies? It would attract a lot of attention I think.
- pfdietz 7mo agoAnd funding dedicated to replication studies.
- MichaelDickens 7mo agoEconomics has the Journal of Comments and Replications in Economics: https://jcr-econ.org/ https://jcr-econ.org/
- LargeWu 7mo agoIs there a viable career path for researchers who choose to focus on replication instead of novel discoveries? I assume replications are perceived as less prestigious, but it's also important work.
- stanford_labrat 7mo agosadly no, this is not a thing and it's critically needed. top on my list of things to do if i were a billionaire: launch an institute for the sole purpose of reproducing other's findings.
- cvwright 7mo agoThe closest thing we have is, in security / privacy / cryptography, you can write "attack" papers. It's not perfect. You don't get any credit unless you can demonstrate a substantial break of the prior work. But it's better than in a lot of other fields.
- fc417fc802 7mo agoWe already have archival journals. What's missing is funding and any prospect of career advancement.
- CGMthrowaway 7mo agoSo much of this started with the rise of the peer-review journal cartel, beginning with Pergamon Press in 1951 (coincidentally founded by Ghislaine Maxwell's father). "Peer review" didn't exist before then, science papers and discussion was published openly, and scientists focused on quality not quantity.
- john_strinlai 7mo ago>Pergamon Press in 1951 (coincidentally founded by Ghislaine Maxwell's father) perhaps a bit off-topic, but what is coincidental about this and/or what is the relevance of Ghislaine Maxwell here?
- benterix 7mo agoIt's useless, but I'm ashamed to admit I found this tiny piece of trivia interesting.
- readthenotes1 7mo agoLike the paywall blocking many scientific arti6, perhaps it would be best if we released also the Epstein Files?
- anonymars 7mo agoI imagine it's the interesting peculiarity that the same people seem to crop up over and over and over again. Six degrees of Kevin Bacon or something, except it's like one or two degrees. As George Carlin said, "it's a big club, and you ain't in it" For example Donald Barr (father of twice-former US Attorney General Bill Barr) hiring college-dropout Jeffrey Epstein whilst headmaster at the elite Dalton School Additional fun facts about Donald Barr: he served in US intelligence during WWII, and wrote a sci-fi book featuring child sex slaves
- jl6 7mo agoAlso the Epstein-Barr virus causes Mono, the clone of .NET, which was created by Bill Gates, known associate of Epstein, whose father was president of the Washington State Bar Association. And you know who else works in Washington? Join the dots, people.
- leoc 7mo agoRight, it seems that many of the weaknesses in the system exist because they serve the interests of journal publishers or of normal, legitimate-ish researchers, but in the process open the door to full-time system-hackers and pure fraudsters.
- bonoboTP 7mo agoAny system that grows too fast has these kinds of problems. When it's a small intimate circle where everyone knows everyone, reputation alone can keep people in check. Once it's larger you need to invent rules and bureaucracies and structures and you will have loopholes that bad actors can more easily exploit, hiding in the crowd, than in the small version. It's the same with the Internet or computing. Security was much less of a topic when it was mostly honest academic nerds using the Internet, and the protocol designs often didn't even assume adversarial participants. Science also still runs on this assumed honesty system that worked well when it was small.
- leoc 7mo agoYes, as long as (for example) you have a huge number of people desperate to claim one or more of a very small number of slots for papers in a few prestige journals then inevitably there must be either a cattle-call selection process, or blatant preference for existing insiders, or both. But things like the fancy journals' reluctance to acknowledge or care about errors in the papers they do publish really aren't at all a necessary consequence of a high submission ratio or of a rapid growth in submissions. In fact it's not even clear that they are a consequence of those pressures at all, let alone that they do anything to mitigate them. They're just things which serve the interests of insiders while in fact making the consequences of the Darwinian pressures worse.
- ramraj07 7mo agoDo you want issues of Nature and cell to be replication studies? As a reader even from within the field, im not interested in browsing through negative studies. It'll be great if I can look them up when needed but im not looking forward to email ToC alerts filled with them. Also who's funding you for replication work? Do you know the pressure you have in tenure track to have a consistent thesis on what you work on? Literally every single know that designs academia is tuned to not incentivize what you complain about. Its not just journals being picky. Also the people committing fraud aren't ones who will say "gosh I will replicate things now!" Replicating work is far more difficult than a lot of original work.
- zhdc1 7mo ago> Do you want issues of Nature and cell to be replication studies? As a reader even from within the field, im not interested in browsing through negative studies. Actually, yes, I do. The marginal cost for publishing a study online at this point is essentially nil.
- recursivecaveat 7mo agoI think archives with pretty low standards for notability are a good idea. At some point though you have to pick what actually counts as interesting enough to go in the curated list that is actually suggested reading, where the prestige is attached. If there's no curation by Nature then it falls to bloggers or another journal to sift through the fire-hose and make best-of lists. Most of the value is in the curation, not the publishing. Without exclusivity there's very little signal.
- mmooss 7mo ago> The marginal cost for publishing a study online at this point is essentially nil. The marginal cost for doing a study remains the same, which is quite a bit. Society doesn't have unlimited scientific talent or hours. Every year someone spends replicating is a year lost to creating something new and valuable.
- Bratmon 7mo ago> Do you want issues of Nature and cell to be replication studies? More than anything. That might legitimately be enough to save science on its own.
- dheera 7mo agoMainstream journals are complicit, but are not the biggest problem. The biggest problem by far is modern society: Tenure, getting paid a livable wage as a researcher, not getting stack-ranked and eliminated from your organization all overindex on positive research results that are marketable. This "loss function" encourages scientific fraud of sorts.
- bonoboTP 7mo agoWhen, in those mythical non-"modern" times, was it easy to get tenure or a livable wage as a researcher? How open were the doors to this and what proportion of society got a realistic chance to pursue such a career? More people getting a chance means more fierce competition.
- canjobear 7mo agoThis isn’t about honest researchers resorting to fraud to publish their null results because they were blocked by big bad Nature. It’s about journals and authors churning out pure junk papers whose only goal is to game metrics like citation count.
- godelski 7mo ago> Most will refuse to publish replications, negative studies, or anything they deem unimportant, even if the study was conducted correctly. I think this was really caused by the rise of bureaucracy in academia. Bureaucrats favorite thing is a measurement, especially when they don't understand its meaning. There's always been a drive for novelty in academia, it's just at the very core of the game. But we placed far too much focus on this, despite the foundation of science being replication. We made a trade, foundation for (the illusion of) progress. It's like trying to build a skyscraper higher and higher without concern for the ground it stands on. Doesn't take a genius to tell you that building is going to come crashing down. But proponents say "it hasn't yet! If it was going to fall it would have already" while critics are actually saying "we can't tell you when it'll fall, but there's some concerning cracks and we're worried it'll collapse and we won't even be able to tell we're in a pile of rubble." I don't know what the solution is, but I do know that our fear of people wasting money and creating fraudulent studies has only resulted in wasting money and fraudulent studies. We've removed the verification system while creating strong incentives to cheat (punish or perish, right?). I think one thing we do need to recognize is that in the grand scheme of things, academia isn't very expensive. A small percentage of a large number is still a large number. Even if half of academics were frauds it would be a small percentage of waste, and pale in comparison to more common waste, fraud, and abuse of government funds. From what I can tell, the US spent $60bn for University R&D in 2023[0] (less than 1% of US Federal expenditures). But in that same time there was $400bn in waste and fraud through Covid relief funds [1]. With $280bn being straight up fraud. That alone is more than 4x of all academic research funding!!! I'm unconvinced most in academia are motivated by money or prestige, as it's a terrible way to achieve those things. But I am convinced people are likely to commit fraud when their livelihoods are at stake or when they can believe that a small lie now will allow them to continue doing their work. So as I see it, the publish or perish paradigm only promotes the former. The lack of replication only allows, and even normalizes, the latter. The stress for novelty only makes academics try to write more like business people, trying to sell their product in some perverse rat race. So I think we have to be a bit honest here. Even if we were to naively make this space essentially unregulated it couldn't be the pinnacle of waste, fraud, and abuse that many claim it is. But I doubt even letting scientists be entirely free from publication requirements that you'd find much waste, fraud, and abuse. Science has a naturally regulating structure. It was literally created to be that way! We got to where we are in through this self regulating system because scientists love to argue about who is right and the process of science is meant to do exactly that. Was there waste and fraud in the past? Yes. I don't think it's entirely avoidable, it'll never be $0 of waste money. But the system was undoubtably successful. And those that took advantage of the system were better at fooling the public than they were their fellow scientists. Which is something I think we've still failed to catch onto [0] https://usafacts.org/articles/what-do-universities-do-with-the-billions-they-receive-from-the-government/ https://usafacts.org/articles/what-do-universities-do-with-t... [1] https://apnews.com/article/pandemic-fraud-waste-billions-small-business-labor-fb1d9a9eb24857efbe4611344311ae78 https://apnews.com/article/pandemic-fraud-waste-billions-sma...
- bonoboTP 7mo agoPlenty will publish it, but those are not as highly regarded by the community. It's not a problem of journals. It's not hard to start your own journal by teaming up with other academics. In machine learning, ICLR is such a venue for example. The problem is much deeper and more fundamental. You want to publish alongside groundbreaking novel research. Researcher's own ears perk up when they hear about something new. They invite colleagues to talk about their novel discoveries not to describe all their null results and successful replications of known results. Funding agencies want research with novelty and impact. They want to write reports to the higher ups and the politicians and the donors that document the innovations that their funding brought. The media will republish press releases that have cool new results. To have research happening, you need someone saying "I want to give money to this researcher". There is an endless queue of people lining up who are ready to take this money and do something with it. The person with money (govt or private) has to use some heuristics to pick. One way is to say "I trust this one, I don't care too much what the project is, I'm sure this person will do something that makes sense". But that is dependent on a track record.
- obviouslynotme 7mo agoI have worked in this particular sausage factory. Multiple funded random replications are the only thing that will save science from this crisis. The scientific method works. We need to actually do it. Replications don't have to be in the journals either. As long as money flows, someone will do them, and that is what matters. The randomization will help prevent coordination between authors and replicators. In a better world, negative studies and replications would count towards tenure, but that is unlikely to occur. At least half of the problem is the pressure to continuously publish positive results.
- bonoboTP 7mo agoRegardless of what gets taught in school about science being objective and without ego, or having a culture of adversarial checks on each other etc., the reality is that scientists are humans and have egos and have petty feuds. Publishing a failed replication of the work of a colleague will not earn you many brownie points. I'm stating this as an observation of what is the case, not as something that I think should be the case. If you attack other researchers like this and damage their reputation - even if for valid scientific reason - you'll have a hard time when those colleagues sit on committees deciding about your next grant etc. Of course if you discover something truly monumental that will override this. But simply sniping down the mediocre research published by other run-of-the-mill researchers will get you more trouble than good. Yes it's directly in contradiction to the textbook-ideal of what science should be, as described to high school students, but there are many things in life this way. Of course it can be laudable to go on such a crusade despite all this, and to relentlessly pursue scientific truth, etc. but that just won't scale.
- obviouslynotme 7mo agoYou are absolutely correct. Even distributing the replication around the world will only help so much. It's a small world out there and only smaller in the specializations. That's why replication has to be required and standard. It will hurt to tear off the bandaid, but once the culture shifts, people will hesitate to publish mediocre research in the first place. Without mediocre research flooding the zone, real numbers will dominate and inflated expectations will wither.
- baxtr 7mo agoOne major contributing factor, in my opinion, is that almost no one in the community was taught the scientific method / epistemology itself. The simple fact that theories should be falsified and not verified is something that most scientists don’t know.
- charlieyu1 7mo agoThe publish or perish culture leads to this. The businessmen and politicians should never be allowed to decide academic fundings.