10 ms·
The entire research field needs major reform urgently. There are so many corrupting incentives for its practitioners that nearly most research findings (at leas
by drgo 2y ago
The entire research field needs major reform urgently. There are so many corrupting incentives for its practitioners that nearly most research findings (at least in my field: medicine) are suspect. Most researchers have neither the time nor energy to work on the important problems, to perfect their craft and to understand the limitations of the scientific methods. The metrics used in evaluating grant applications and academic promotion push researchers to work on short-term sensationalist and easy-to-publish ideas. This is sad because if there was a time we need solid unimpugnable science is now.
- hedora 2y agoI think the scale of the problem varies wildly between different fields of science. For instance, in computer science, double blind reviewing is the norm, and the biggest issue is that there is an incentive to produce more papers with smaller contributions, instead of a mixture of larger, better, papers, and 1-2 column notes that describe important, but simple results (such notes used to be published routinely). In other fields, not only are submissions not anonymous, but editors sometimes override the reviewers (in both directions), for what appear to be political purposes.
- Der_Einzige 2y agoUhh, the elephant in the room for CS/ML publications is that a lot of people hardcore lie on their results page since almost no one tries to reproduce anything. I've witnessed folks straight out of their undergrad with literally 1 workshop paper at an ACL track conference get thrown 150K starting offers by no-name startups. Imagine how important it must be to get your first NeurIPS main conference publication. The ROI likely amounts to 300K+ per year, so if anything, I'm surprised that cheating/lies in the academy aren't even more brazen. Dynamics like this are why both Stanford and Harvard's presidents had to step-down for gross academic plagerism/misconduct. 1. https://www.science.org/doi/10.1126/science.359.6377.725 https://www.science.org/doi/10.1126/science.359.6377.725 (2018) 2. https://www.technologyreview.com/2020/11/12/1011944/artificial-intelligence-replication-crisis-science-big-tech-google-deepmind-facebook-openai/ https://www.technologyreview.com/2020/11/12/1011944/artifici... (2020) 3. https://www.nature.com/articles/d41586-023-03817-6 https://www.nature.com/articles/d41586-023-03817-6 (2023, Nature)
- godelski 2y agoI've seen it happen even when source is available. I've also seen a lot of really terrible math mistakes in ML because people don't understand the underlying assumptions of the metrics they use. The rot is everywhere because people forgot the goal. We all get caught up in the metrics that are used to judge us that we forgot what we were proxying in the first place.
- jltsiren 2y agoDouble blind reviewing is not the norm in computer science. You will sometimes see it, but it depends on the subfield, the specific conference/journal, and even the year. In my experience as an author/reviewer, the submissions have been anonymous maybe 10% of the time. And the biggest issue I see in CS is the focus on gatekeeping. When many (most?) papers are published in conferences with a single round of reviews, peer reviewing becomes more about accepting/rejecting the paper than improving it. The average review I receive in bioinformatics is more useful to me than the average review in CS.
- ilrwbwrkhv 2y agoYes, you are absolutely right, and this in turn has led to people disbelieving science and things which were once a bipartisan issue, such as climate change, has now been termed a conspiracy by people on the right. This has only happened because people do not trust scientists and science anymore.
- wakawaka28 2y agoWhat if science and politics really are corrupt enough to corroborate the conspiracy theories? We need an International Journal of Conspiracy Studies right now!
- Mountain_Skies 2y agoFunny how the unabashedly partisan are the ones who make partisan statements in the same breath as their denouncements of partisanship.
- Ar-Curunir 2y agoDeclining trust in science is not because of lack of open-access publications. The right is not ignoring research because it is behind a paywall or because it hasn’t been reproduced. In fact, the right has chosen to willingly ignore research that has been painstakingly reproduced, all in service of the god of greed and ideology.
- DataDive 2y agoIn my opinion the declining trust is science is due to science being far less apolitical and far less objective and less validated than in the past.
- Ar-Curunir 2y agoWhich era of science was apolitical, exactly? The one where Galileo was prosecuted by the church? Or one where the US military invested billions of dollars into early computer science? Or one where nearly every physicist was staunchly anti-proliferationist ? Science has not, and will never be, apolitical. It is just that you are noticing that the politics of some scientists does not align with yours or with some politicians, and so you’re noticing the difference.
- aborsy 2y agoIt has become a game of reputation, citations and the number of papers. It works the same way that a company does, with the actual work being done by students and postdocs and citations accumulated by management (a lot of them don’t do research, rather spend time putting their names in different papers, collecting more and more students, and playing academic politics). After a while, they become a luminary and collect awards.
- nine_k 2y agoIt looks eerily similar to the problems of businesses that happen when all what the executives care about is the stock price growth in the current quarter.
- EvgeniyZh 2y agoWhat kind of reform?
- paretoer 2y agoI don't personally see a solution other than student loan money being much harder to get. There is so much bad research because there are too many researchers. For myself, I think I am smart enough to have worked on a PhD in economics but not smart enough to have done anything worthwhile or to have advanced the field. I am certain I would not have. Then in the face of this massive student loan debt I would have produced junk research and try to game the system to make the investment worthwhile. The incentives are all ridiculous in the current system for most. The idea politicians will cut student loan money in order to produce better scientific research would be a good way to optimize for losing an election. Personally, I don't think there is a way out. If anything it probably just gets worse.
- EvgeniyZh 2y agoThere are countries where student loans are not a thing and the science there arguably faces the same problem. PhD is not paid by a student even in US, it's all scholarships.
- sova 2y agoGet paid to answer important questions instead of getting paid to provide sought-after results. I think that's the end-point, the "how we get there" is now the question!
- rscho 2y agoIn clinical medicine, the system needs no reform, because almost nobody's goal is to do research. Clinical research is pretty unique in that it serves almost entirely hierarchical advancement rather than science. Be honest, when was last time you saw a clinical researcher formulate a consistent research hypothesis and constitute a solid project around it? Almost all projects are basically: 'I'll test that with homemade shitty stats because it's trendy!' So IMO, the first reform to conduct is to turn clinical researchers into real scientists. That won't be easy.
- kstrauser 2y agoI was told long ago that the first rule of evaluating a medical treatment is to ignore all the clinical studies because nothing useful comes out of them. I don’t know if that’s actually the case but that seems to be the prevailing thought. Which is awful, because it should be the opposite. Clinical studies should be the gold standard. They’re often conducted by people who are better doctors than scientists though. A zero-blind trial of n=4 might perhaps not be the definitive study the doctor had hoped. (Not to pick on doctors. My wife’s one. Software developers often fall into the same common “I’m good at this one hard in-demand skill so I must be good at all of them” trap.)
- p00dles 2y agoTo ignore _all_ clinical studies because “nothing useful comes out of them” seems extreme. Yes, there are poorly run studies, and conclusions can be inappropriately drawn from studies (ex: when a study wasn’t meant to answer certain questions). However, well-run randomized control trials (RCTs) in medicine absolutely exist and can be a very strong form of evidence when interpreted correctly. This is especially true in comparison to other disciplines (economics, etc.) where such RCT trials are impossible and researchers wish that they a research tool as valuable as an RCT.
- kstrauser 2y agoFair. It was mainly in the context of “clinical studies show that unlikely thing X cures Y”. No randomized double-blind trials? Sure.
- sa-code 2y agoWhat are the current corrupting incentives? And what would an ideal research world look like?