4 ms·

Science is self-correcting, but sometimes, the correction mechanism it is very slow and inconsequential enough that people are incentivized to just publish anyt

by jarenmf 1mo ago

Science is self-correcting, but sometimes, the correction mechanism it is very slow and inconsequential enough that people are incentivized to just publish anything. Peer review is merely a quality check—and not a very good one. Just the other day, I ran into a paper in my field that claims to have solved a central problem that scientists have been working on for 50 years, single-authored, using some fancy mathematics. I was excited to read it, but it didn't take long for me to find out that it's complete BS-derivations that don't follow, made-up equation terms, and conflation of basic terms in the field. I was really surprised that this passed the editor and at least two peer reviewers. I dug deeper and found that the same author has published 11 research articles in top journals in 2026 alone — mostly physics and chemistry, and adjacent fields— and I'm talking about really top journals like Scientific Reports and Physics Letters B. Most of these have serious mathematical flaws that are unfixable. I wrote to a couple of editors, but only one took it seriously enough to initiate an investigation. I made a promise to myself that if no real action is taken, I will seriously consider changing careers.

Yokolos 1mo ago
Even as imperfect as it is, it's infinitely better than the alternatives like belief in religion/religious texts/pastors or anybody else who is just making shit up based on whatever they think sounds good. None of that is falsifiable or makes any attempt to iteratively arrive at anything true or real. There is literally nothing better that we have other than the scientific method. Yes, science is self-correcting, even if it is slow or whatever else. Nothing else that we have can approach even that bar.
dilawar 1mo ago
> Science is self-correcting, but sometimes As long as people mislead or make mistake unknowingly, it's fine to rely to self-correcting mechanism of science. The institutional rot is much deeper. It is not good to leave all this fraud to self-correcting mechanisms of science. We also have "trust" to worry about.
- anthonyrstevens 1mo ago
  Maybe instead of throwing around ideas of "institutional rot" we can realize that there are hundreds of thousands of people doing "science" in the right way, are not out for material gain, do not engage in fraud, and are interested in truth and scientific advancement. The fact that there are a vanishingly small # of cases where bad things happen in the process are the exceptions that prove the rule, IMHO. Certain segments of society like to hype these up (or even lie about their frequency) in order to discredit the scientific method. "Institutional rot" is a meme that needs to go away. Identify the problems at the edges, fix them, and move on to letting the bulk of the work continue as it always has - quietly building upon previous valid results to the betterment of society.
stackghost 1mo ago
>I made a promise to myself that if no real action is taken, I will seriously consider changing careers. Personally I think you should name and shame the author as well as the publications and the names of the peer "reviewers" who clearly did sweet-fuck-all.
- Ekaros 1mo ago
  Not to forget the editors responsible at time. Why not leadership and board members if they can be found. Making responsible people responsible publicly is most effective way. Fully deserved shame can do wonders to world.
rvba 1mo ago
Would be nice if you kepy ypur career and created some magazine / journal that focuses on checks and pointing out flaws. That cheaters would dread being included there (would probably be a thick journal too)
- mike_hearn 1mo ago
  They wouldn't dread being included there, not at all. The institutions don't care about fraud and reflexively defend it, ignore it or shoot the messenger. Look at the Arday nonsense. People are still discovering new lies he was telling but Cambridge went to the mat for him. Dan Ariely in particular has faced convincing fraud accusations for years: https://en.wikipedia.org/wiki/Dan_Ariely#Controversies_and_criticism https://en.wikipedia.org/wiki/Dan_Ariely#Controversies_and_c... Nothing ever happens. Universities do not care.
  - stackghost 1mo ago
    The real power move would be to start targeting prospective students. Universities will start caring of enrolment goes down.
    - mike_hearn 1mo ago
      They might not care even then. Journalism is an interesting case study. Their revenues went into sharp decline around the start of the century, dropping by more than 50% in some cases. Concurrently polls showed trust in journalism was plummeting, that people were tuning out because they felt that what they read wasn't reliable. Journalism didn't solve this problem by rebuilding trust and their audience. Instead they started reforming as non-profit institutions and taking huge payments from philanthropists to push specific agendas, without alerting readers to that fact. https://apnews.com/article/science-business-arts-and-entertainment-journalism-united-states-087d1d5dd7189c529fe5d7a21a1ffb5f https://apnews.com/article/science-business-arts-and-enterta... These grants aren't tied to readership but rather to the content of what's published, so they can produce lots of articles even if nobody reads them. Those articles are then used as "reliable sources" to control Wikipedia, and from there what search engines and LLMs say. If students start checking out at scale (already happening), or if governments cut funding (already happening in the USA), their most likely move is to just replace the revenue with money from the Gates Foundation et al, and keep going just as they are. Their primary role will switch from education or research to the propagation of approved narratives.
mike_hearn 1mo ago
Science isn't self correcting. This is a harmful meme that we need to collectively put in the trash can, as it's constantly used by academics to evade accountability. The core problem is people conflating science with Science™. The scientific method is a set of techniques that help people discover and report true things. But there are two huge problems with it: 1. What exactly is the scientific method? It's not like there's a constitutional amendment that lays it out. Try and pin people down and you'll discover nobody agrees on what science actually requires. 2. Even if some subset of people did write down a definition, nobody is checking anyone is following it. So people conflate science the Platonic methodological ideal with Science™, the institutions that claim most forcefully to be its avatar. But those institutions have neither a definition nor any enforcement arm, so fraud and abuse runs rampant. There's no reason why anything should self correct in this environment outside of corporate science, where the self-correction mechanism is managerial intervention or, in the absence of that, bankruptcy.
- ahartmetz 1mo ago
  3. Publish or perish, scientific publications (count x impact factor) as careeer credits. What a terrible yet hard to kill idea.
- aktenlage 1mo ago
  My take on the self correction is, that later research usually cannot be build on wrong results. So over the decades or centuries the correct results prevail.
  - mike_hearn 1mo ago
    You'd think so. I used to believe this too. Surely you try and build on a result which creates an implicit replication study, then your experiments don't work and the problem is discovered pretty quick. Hence, self correction. It often doesn't happen. The reasons seem to vary by field. We do see it in a few very hard fields like materials science. In the softer end of social sciences like anthropology, education or the humanities it's obvious why not: they don't make empirically testable claims about reality anyway so there's nothing to replicate. This is the Arday problem, where he was publishing what they call auto-ethnography: effectively just blog posts about the author's own feelings. He even interviewed himself in the third person. In the harder social sciences it's because nothing actually builds on anything else (see "A problem in theory" [1]). Psychology and related fields like sociology, criminology, etc are basically just a pile of random ideas pooped out in brainstorming sessions. There are hardly any overarching theories generating testable predictions, so studies test whatever random claims the researcher thinks is interesting enough to get in the news. Everything is just rocks lying on the ground rather than towers of understanding, so when one paper is overturned it has no impact on anything else. Citation counts obscure this fact, but do some reverse-citation checks and you'll find a lot of citations are worthless or outright invalid. In modeling based fields like economics, epidemiology or climatology, it's because they can't put the thing they study in a lab and do high speed experimentation. "Replication" comes to mean repeating the same analysis on the same data set. Often they literally just run the same program on the same input file, get the same result and call it replicated - or sometimes they call it replicated even if the outputs don't match! See [2]. But the program is just a set of assumptions, not real discoveries, so building more papers on top of that program's output doesn't detect invalidity. It will all replicate in some very narrow technical sense without that proving anything useful. Sometimes outsiders notice what's happening but over time academia has built up a complex web of lore that protects them psychologically. For example, they routinely reject or ignore feedback from outside their field on the basis that it doesn't come from "experts", meaning themselves, or it comes from "right wing" people, meaning anyone who isn't an academic or close political ally. [1] https://www.nature.com/articles/s41562-018-0522-1 https://www.nature.com/articles/s41562-018-0522-1 [2] https://dailysceptic.org/2020/05/06/code-review-of-fergusons-model/ https://dailysceptic.org/2020/05/06/code-review-of-fergusons...
Otterly99 1mo ago
I have been having the opposite problem with my papers lately. Instead of glossing over mistakes, the reviewer report missing information which is clearly stated in the text. This is quite frustrating and hard to stay polite after the fifth "please just read the text." I really wish there was some serious rework of the publishing system. Nobody is happy about it but nothing seems to be moving.
HPsquared 1mo ago
It's a bit like how people believe markets are self-correcting. Yes, only over an infinite time horizon, in theory. Edit: and like chemical equilibrium, you could say it depends on the "rate of wrongness production" compared to the "rate of correction". Edit edit: self-correction of course must fight against the higher virality of work selected for its impact rather than correctness, and incorrect work can then spawn further incorrect work, like an autocatalytic process or epidemic.
Metacelsus 1mo ago
>really top journals like Scientific Reports I wouldn't consider Scientific Reports to be a top journal, it's basically pay-to-publish
deleted 1mo ago
[deleted]
iLoveOncall 1mo ago
And you protect them by not sharing their name.

by jarenmf 1mo ago

Yokolos 1mo ago

Even as imperfect as it is, it's infinitely better than the alternatives like belief in religion/religious texts/pastors or anybody else who is just making shit up based on whatever they think sounds good. None of that is falsifiable or makes any attempt to iteratively arrive at anything true or real. There is literally nothing better that we have other than the scientific method. Yes, science is self-correcting, even if it is slow or whatever else. Nothing else that we have can approach even that bar.

dilawar 1mo ago

> Science is self-correcting, but sometimes As long as people mislead or make mistake unknowingly, it's fine to rely to self-correcting mechanism of science. The institutional rot is much deeper. It is not good to leave all this fraud to self-correcting mechanisms of science. We also have "trust" to worry about.

anthonyrstevens 1mo ago
Maybe instead of throwing around ideas of "institutional rot" we can realize that there are hundreds of thousands of people doing "science" in the right way, are not out for material gain, do not engage in fraud, and are interested in truth and scientific advancement. The fact that there are a vanishingly small # of cases where bad things happen in the process are the exceptions that prove the rule, IMHO. Certain segments of society like to hype these up (or even lie about their frequency) in order to discredit the scientific method. "Institutional rot" is a meme that needs to go away. Identify the problems at the edges, fix them, and move on to letting the bulk of the work continue as it always has - quietly building upon previous valid results to the betterment of society.

stackghost 1mo ago

>I made a promise to myself that if no real action is taken, I will seriously consider changing careers. Personally I think you should name and shame the author as well as the publications and the names of the peer "reviewers" who clearly did sweet-fuck-all.

Ekaros 1mo ago
Not to forget the editors responsible at time. Why not leadership and board members if they can be found. Making responsible people responsible publicly is most effective way. Fully deserved shame can do wonders to world.

rvba 1mo ago

Would be nice if you kepy ypur career and created some magazine / journal that focuses on checks and pointing out flaws. That cheaters would dread being included there (would probably be a thick journal too)

mike_hearn 1mo ago
They wouldn't dread being included there, not at all. The institutions don't care about fraud and reflexively defend it, ignore it or shoot the messenger. Look at the Arday nonsense. People are still discovering new lies he was telling but Cambridge went to the mat for him. Dan Ariely in particular has faced convincing fraud accusations for years: https://en.wikipedia.org/wiki/Dan_Ariely#Controversies_and_criticism https://en.wikipedia.org/wiki/Dan_Ariely#Controversies_and_c... Nothing ever happens. Universities do not care.
- stackghost 1mo ago
  The real power move would be to start targeting prospective students. Universities will start caring of enrolment goes down.
  - mike_hearn 1mo ago
    They might not care even then. Journalism is an interesting case study. Their revenues went into sharp decline around the start of the century, dropping by more than 50% in some cases. Concurrently polls showed trust in journalism was plummeting, that people were tuning out because they felt that what they read wasn't reliable. Journalism didn't solve this problem by rebuilding trust and their audience. Instead they started reforming as non-profit institutions and taking huge payments from philanthropists to push specific agendas, without alerting readers to that fact. https://apnews.com/article/science-business-arts-and-entertainment-journalism-united-states-087d1d5dd7189c529fe5d7a21a1ffb5f https://apnews.com/article/science-business-arts-and-enterta... These grants aren't tied to readership but rather to the content of what's published, so they can produce lots of articles even if nobody reads them. Those articles are then used as "reliable sources" to control Wikipedia, and from there what search engines and LLMs say. If students start checking out at scale (already happening), or if governments cut funding (already happening in the USA), their most likely move is to just replace the revenue with money from the Gates Foundation et al, and keep going just as they are. Their primary role will switch from education or research to the propagation of approved narratives.

mike_hearn 1mo ago

Science isn't self correcting. This is a harmful meme that we need to collectively put in the trash can, as it's constantly used by academics to evade accountability. The core problem is people conflating science with Science™. The scientific method is a set of techniques that help people discover and report true things. But there are two huge problems with it: 1. What exactly is the scientific method? It's not like there's a constitutional amendment that lays it out. Try and pin people down and you'll discover nobody agrees on what science actually requires. 2. Even if some subset of people did write down a definition, nobody is checking anyone is following it. So people conflate science the Platonic methodological ideal with Science™, the institutions that claim most forcefully to be its avatar. But those institutions have neither a definition nor any enforcement arm, so fraud and abuse runs rampant. There's no reason why anything should self correct in this environment outside of corporate science, where the self-correction mechanism is managerial intervention or, in the absence of that, bankruptcy.

ahartmetz 1mo ago
3. Publish or perish, scientific publications (count x impact factor) as careeer credits. What a terrible yet hard to kill idea.
aktenlage 1mo ago
My take on the self correction is, that later research usually cannot be build on wrong results. So over the decades or centuries the correct results prevail.
- mike_hearn 1mo ago
  You'd think so. I used to believe this too. Surely you try and build on a result which creates an implicit replication study, then your experiments don't work and the problem is discovered pretty quick. Hence, self correction. It often doesn't happen. The reasons seem to vary by field. We do see it in a few very hard fields like materials science. In the softer end of social sciences like anthropology, education or the humanities it's obvious why not: they don't make empirically testable claims about reality anyway so there's nothing to replicate. This is the Arday problem, where he was publishing what they call auto-ethnography: effectively just blog posts about the author's own feelings. He even interviewed himself in the third person. In the harder social sciences it's because nothing actually builds on anything else (see "A problem in theory" [1]). Psychology and related fields like sociology, criminology, etc are basically just a pile of random ideas pooped out in brainstorming sessions. There are hardly any overarching theories generating testable predictions, so studies test whatever random claims the researcher thinks is interesting enough to get in the news. Everything is just rocks lying on the ground rather than towers of understanding, so when one paper is overturned it has no impact on anything else. Citation counts obscure this fact, but do some reverse-citation checks and you'll find a lot of citations are worthless or outright invalid. In modeling based fields like economics, epidemiology or climatology, it's because they can't put the thing they study in a lab and do high speed experimentation. "Replication" comes to mean repeating the same analysis on the same data set. Often they literally just run the same program on the same input file, get the same result and call it replicated - or sometimes they call it replicated even if the outputs don't match! See [2]. But the program is just a set of assumptions, not real discoveries, so building more papers on top of that program's output doesn't detect invalidity. It will all replicate in some very narrow technical sense without that proving anything useful. Sometimes outsiders notice what's happening but over time academia has built up a complex web of lore that protects them psychologically. For example, they routinely reject or ignore feedback from outside their field on the basis that it doesn't come from "experts", meaning themselves, or it comes from "right wing" people, meaning anyone who isn't an academic or close political ally. [1] https://www.nature.com/articles/s41562-018-0522-1 https://www.nature.com/articles/s41562-018-0522-1 [2] https://dailysceptic.org/2020/05/06/code-review-of-fergusons-model/ https://dailysceptic.org/2020/05/06/code-review-of-fergusons...

Otterly99 1mo ago

I have been having the opposite problem with my papers lately. Instead of glossing over mistakes, the reviewer report missing information which is clearly stated in the text. This is quite frustrating and hard to stay polite after the fifth "please just read the text." I really wish there was some serious rework of the publishing system. Nobody is happy about it but nothing seems to be moving.

HPsquared 1mo ago

It's a bit like how people believe markets are self-correcting. Yes, only over an infinite time horizon, in theory. Edit: and like chemical equilibrium, you could say it depends on the "rate of wrongness production" compared to the "rate of correction". Edit edit: self-correction of course must fight against the higher virality of work selected for its impact rather than correctness, and incorrect work can then spawn further incorrect work, like an autocatalytic process or epidemic.

Metacelsus 1mo ago

>really top journals like Scientific Reports I wouldn't consider Scientific Reports to be a top journal, it's basically pay-to-publish

deleted 1mo ago

[deleted]

iLoveOncall 1mo ago

And you protect them by not sharing their name.