9 ms·

Why Most Published Research Findings Are False (2005)

231 points by Michelangelo11 2y ago

codingwagie 2y ago
[flagged]
- JohnMakin 2y ago
  That’s a lot of mental hoops you just leaped through. Are you saying you think immigration policy is the main contributing cause of poor research quality? Because that is a wild claim without evidence. Also doesn’t really make sense, immigration policy varies from state to state and there isn’t just one single country producing research.
  - codingwagie 2y ago
    Some huge percentage of STEM graduate school is just for immigration These people are not seeking "truth"
    - lIllIlIIlIllIlI 2y ago
      https://en.wikipedia.org/wiki/No_true_Scotsman https://en.wikipedia.org/wiki/No_true_Scotsman
    - worik 2y ago
      Evidence? Especially given the context.
      - adamrezich 2y ago
        What kind of evidence would sway your belief on this matter? Perhaps a published academic research paper on the topic?
        worik 2y ago
        > What kind of evidence would sway your belief on this matter? I do not have beliefs on this matter, I am interested in data. I would like to know what studies of graduate student intentions and outcomes > Perhaps a published academic research paper on the topic? I see what you (think you are) doing there. That would be a start
        adamrezich 2y ago
        It would be start, for people like you who hopelessly cling to your deference to the authority that academic research papers exert over people like yourselves. For the rest of us, it would be meaningless.
        worik 2y ago
        > For the rest of us, I am curious: What are your decision criteria? Academic research papers are systematically flawed, true. But they form a part of the story of finding truth. What do you use?
    - spicybright 2y ago
      Aren't they required to have previous schooling in their home country, pass an aptitude test against others, and maintain good grades to stay in the program in order to stay in the country? I don't see how you can avoid actually doing the education part here. And I'm sure lots see immigration + better education as a win/win, which I don't see a problem with. Where are you getting the huge percentage from? Do you have sources for your claim? Even news articles?
    - cortesoft 2y ago
      The numbers I find say about 20% of grad students in the United States are international students. I am not sure that is a huge percentage?
      - adamrezich 2y ago
        What does that statistic look like for other nations?
    - jjulius 2y ago
      >These people are not seeking "truth" Careful with those absolutes, there. And the whole "these people" thing, too, probably.
- kimixa 2y ago
  In 2022 there were ~57k PHDs awarded in the USA [0]. In the same year, there were ~500k immigrant visas [1]. If every single PHD was for immigration, it'll still only be ~10% of the total. And even if the thesis was "forced", just getting to PHD level means you're very much in the top echelons of education, and even then takes multiple years - as someone who immigrated with "only" a BSc I suspect they'll be able to use one of the many other paths instead with less effort. I'm not sure the logic holds. [0] https://www.forbes.com/sites/michaeltnietzel/2024/02/05/number-of-doctoral-degrees-awarded-in-us-rebounds-to-all-time-high/ https://www.forbes.com/sites/michaeltnietzel/2024/02/05/numb... [1] https://www.bal.com/perspectives/bal-news/united-states-us-visa-statistics-show-continued-increase-in-visa-issuance/ https://www.bal.com/perspectives/bal-news/united-states-us-v...
vouaobrasil 2y ago
> In this framework, a research finding is less likely to be true [...] where there is greater flexibility in designs, definitions, outcomes, and analytical modes It's worth noting though that in many research fields, teasing out the correct hypotheses and all affecting factors are difficult. And, sometimes it takes quite a few studies before the right definitions are even found; definitions which are a prerequisite to make a useful hypothesis. Thus, one cannot ignore the usefulness of approximation in scientific experiments, not only to the truth, but to the right questions to ask. Not saying that all biases are inherent in the study of sciences, but the paper cited seems to take it for granted that a lot of science is still groping around in the dark, and to expect well-defined studies every time is simply unreasonable.
- 3np 2y ago
  This is only meaningful if "the replicaton crisis" is systematically addressed.
marcosdumay 2y ago
Yeah, when you try new things, you often get them wrong. Why do we expect most published results to be true?
- ekianjo 2y ago
  because people believe that peer review improve things but in fact not really. its more of a stamping process
  - elashri 2y ago
    Yes that a misconception that many people think that peer-review involves some sort of verification or replication which is not true. I would blame mainstream media in part for this and how they report on research and don't emphasize this nature. Mainstream media also is not interested in reporting on progress but likes catchy headlines/findings.
  - deleted 2y ago
    [deleted]
- bluefirebrand 2y ago
  Because people use published results to justify all sorts of government policy, business activity, social programs, and such. If we cannot trust that results of research are true, then how can we justify using them to make any kind of decisions in society? "Believe the science", "Trust the experts" etc sort of falls flat if this stuff is all based on shaky research
  - marcosdumay 2y ago
    > If we cannot trust that results of research are true, then how can we justify using them to make any kind of decisions in society? Well, don't. Make your decisions based on replicated results. Stop hyping single studies.
    - XorNot 2y ago
titanomachy 2y ago
2022
elashri 2y ago
There is at least one thing wrong about this. This is an essay about a paper published a simulation based scenarios in medical research. It then try to generalize to "research" and avoid this very narrow support to the claim. I think this is something true and it should make us more cautious when deciding based on single studies. But things are different in other fields. Also this is called research. You don't know the answer before head. You have limitations in tech and tools you use. You might miss something, didn't have access to more information that could change the outcome. That is why research is a process. Unfortunately common science books talks only about discoveries, results that are considered fact but usually don't do much about the history of how we got there. I would like to suggest a great book called "How experiments end"[1] and enjoy going into details on how scientific conscious is built for many experiments in different fields (mostly physics). [1] https://press.uchicago.edu/ucp/books/book/chicago/H/bo5969426.html https://press.uchicago.edu/ucp/books/book/chicago/H/bo596942...
- thatguysaguy 2y ago
  I think it's clear that this paper has stood the rest of time over the last 20 years. Our estimates of how much published work fails to replicate or is outright fraudulent have only increased since then.
  - tptacek 2y ago
    Outright research fraud is probably very rare; the cases we've heard about stick out, but people outside of academia usually don't have a good intuition for just how vast the annual output of the sciences are. Remember the famous PhD comic, showing how your thesis is going to be an infinitesimal fraction of the work of your field.
    - dekhn 2y ago
      Is incompetence fraud? Or just incompetence? I'm asking because a fair number of the molecular biologists who get caught by Elizabeth Bik for copy/pasting images of gels insist they just made honest mistakes (with some commentary about the atrocious nature of record-keeping in modern biology). I alter Ionnides's conclusion to be instead: "Roughly 50% of papers in quantitative biological sciences contain at least one error serious enough to invalidate the conclusion" and "Roughly 75% of really interesting papers are missing at least one load-bearing method detail that reproducers must figure out on their own" (my own personal observations of the literature are consistent with these rates; I was always flabbergasted at people who just took Figure 3 as correct).
skybrian 2y ago
(2005). I wonder what's changed?
- youainti 2y ago
  Over on peerpub there has been some discussion of studies on the topic. https://pubpeer.com/publications/14B6D332F814462D2673B6E9EF9AF9#4 https://pubpeer.com/publications/14B6D332F814462D2673B6E9EF9...
motohagiography 2y ago
i wonder if science could benefit from publishing using pseudonyms the way software has. if it's any good, people will use it, the reputations will be made by the quality of contributions alone, it makes fraud expensive and mostly not worth it, etc.
- wwweston 2y ago
  People have uses for conclusions that sometimes don't have anything to do with their validity. So while "if it's any good, people will use it" is true and quality contributions will be useful, the converse is not true: the use or reach of published work may be only tenuously connected to whether it's good. Reputation signals like credentials and authority have their limits/noise, but bring some extra signal to the situation.
  - motohagiography 2y ago
    what's missing from this paper is a probability using its own model that it too is false. counter to the headline, it implies that by its own probable falsehood, most published research is in fact true. I admit to missing the joke in the first reading. pseudonyms may prevent the abuse of invalid papers by removing the ability of the authors to front institutional reputations for partisan claims. the movement for science and data to drive policy outside of their domains sounds nice until you find that the science and data are irrepreducible, and the institutions have become laundering vehicles for debased opinions that wash the hands of policymakers. as though the potential for abuse has become the value. maybe it's a rarefied kind of funny, but the kernel of truth it reveals is that it could be time to start using pseudonyms in some disciplines to make the axis of policymakers and academics more honest.
debacle 2y ago
Most? Really?
ape4 2y ago
So is this paper false too? .. infinite recursion...
- wccrawford 2y ago
  Most probably.
youainti 2y ago
Please note the peerpub comments discussing that it appears that followup research shows about 15% is wrong, not the 5% anticipated. https://pubpeer.com/publications/14B6D332F814462D2673B6E9EF9AF9 https://pubpeer.com/publications/14B6D332F814462D2673B6E9EF9...
carabiner 2y ago
This only applies to life sciences, social sciences right? Or are most papers in computer science or mechanical engineering also false?
- thatguysaguy 2y ago
  It's very bad in CS as well. See e.g.: https://arxiv.org/abs/1807.03341 https://arxiv.org/abs/1807.03341 IIRC there was also a paper analyzing how often results in some NLP conference held up when a different random seed or hyperparameters were used. It was quite depressing.
- pcrh 2y ago
  It's mostly in medicine and psychology. In topics where there is less reliance on relatively small numbers of cases (as is typical for medicine), there is also less reliance on marginal, but statistically "significant", findings. So areas such as biochemistry, chemistry, even some animal studies, are less susceptible to over-interpretation or massaging of data.
- mike_hearn 2y ago
  Life sciences, social sciences, eco-sciences, humanities. Engineering is mostly OK or at least not mostly false. When you get into the humanities a lot of papers "aren't even wrong", as in, the authors don't hold themselves to any standards of logic or rigor to begin with, or aren't even making any kind of identifiable claim about the world, and don't consider rebuttals based on such criticisms to be valid. A lot of people overlook ecology/climatology because millenarians have made it into such a live wire, but those fields generally have worse problems than medicine. For instance, in medicine going back and retroactively changing patient records for your clinical trial is taboo. It happens remarkably often, but, everyone accepts that it's not supposed to. In climatology they retroactively change climate datasets all the time and if anyone calls them out on it they just attack the critics. They don't even accept the principle that their models should predict data as measured using a constant methodology. It's meaningless in such a field to say "the hypothesis is consistent with the data" because the historical data you analyzed might be replaced with a new version that's fundamentally different.
withinboredom 2y ago
I've implemented several things from computer science papers in my career now, mostly related to database stuff. They are mostly terribly wrong or show the exact OPPOSITE as to what they claim in the paper. It's so frustrating. Even occasionally, they offer their code used to write the paper and it is missing entire features they claim are integral for it to function properly; to the point that I wonder how they even came up with the results they came up with. My favorite example was a huge paper that was almost entirely mathematics-based. It wasn't until you implemented everything that you would realize it just didn't even make any sense. Then, when you read between the lines, you even saw their acknowledgement of that fact in the conclusion. Clever dude. Anyway, I have very little faith in academic papers; at least when it comes to computer science. Of all the things out there, it is just code. It isn't hard to write and verify what you purport (usually takes less than a week to write the code), so I have no idea what the peer reviews actually do. As a peer in the industry, I would reject so many papers by this point. And don't even get me started on when I send the (now professor) questions via email to see if I just implemented it wrong, or whatever, that just never fucking reply.
- reasonableklout 2y ago
  Wow, sounds awful. Help the rest of us out - what was the huge paper that didn't work or was actively misleading?
  - withinboredom 2y ago
    I'd rather not, for obvious reasons. The less obvious reason is that I don't remember the title/author of the paper. It was back in 2016/17 when I was working on a temporal database project at work and was searching literature for temporal query syntax though.
- Lerc 2y ago
  For papers with code, I have a seen a tendency to consider the code, not the paper to be the ground truth. If the code works, then it doesn't matter what the paper says, the information is there. If the code doesn't work, it seems like a red flag. It's not an advantage that can be applied to biology or physics, but at least computer science catches a break here.
blackeyeblitzar 2y ago
It’s a matter of incentives. Everyone who wants a PhD has to publish and before that they need to produce findings that align with the values of their professors. These bad incentives combined with rampant statistical errors lead to bad findings. We need to stop putting “studies” on a pedestal.
breck 2y ago
On a livestream the other day, Stephan Wolfram said he stopped publishing through academic journals in the 1980's because he found it far more efficient to just put stuff online. (And his blog is incredible: https://writings.stephenwolfram.com/all-by-date/ https://writings.stephenwolfram.com/all-by-date/) A genius who figured it academic publishing had gone to shit decades ahead of everyone else. P.S. We built the future of academic publishing, and it's an order of magnitude better than anything else out there.
- jordigh 2y ago
  Genius? The one who came up with a new kind of science?
  - breck 2y ago
    Do you think judging someone by your least favorite work of their's is a good strategy? Do you also say, "Newton a genius? The one who tried to turn lead into gold?"
    - jordigh 2y ago
      He still parades that around as his magnum opus in his latest blog post: https://writings.stephenwolfram.com/2024/08/five-most-productive-years-what-happened-and-whats-next/ https://writings.stephenwolfram.com/2024/08/five-most-produc... Yeah, I think it's fair to judge him by it.
      - breck 2y ago
        Fair enough. Like all of his work, I thought it was an incredible book, if you just randomly sample 10% of it. I never understood why he doesn't cut more, as he has genius ideas that get really watered down with lots of less relevant details. I would love if he started doing 1 page tldr's for all of his works.
- wahern 2y ago
  He created his own peer reviewed academic journal and founded a corporation to publish it: https://en.wikipedia.org/wiki/Complex_Systems_(journal) https://en.wikipedia.org/wiki/Complex_Systems_(journal) That's a little different than just putting stuff online.
ants_everywhere 2y ago
This is a classic and important paper in the field of metascience. There are other great papers predating this one, but this one is widely known. Unfortunately the author John Ioannidis turned out to be a Covid conspiracy theorist, which has significantly affected his reputation as an impartial seeker of truth in publication.
- kelipso 2y ago
  Ha how meta is this comment because the obvious inference one makes from the title is "Why Most Published Research Findings on Covid Are False" and that goes against the science politics. If only he had avoided the topic of Covid entirely, then he would be well regarded.
  - ants_everywhere 2y ago
    It is pretty meta I guess. > Why Most Published Research Findings on Covid Are False Well, that's why there was so much focus on replication, multiple data sources and meta-analyses. The focus was there because the assumption is each study is flawed and those tools help extract better signal from the noise of individual studies. > and that goes against the science politics I don't think I follow you here. Are you referring to the anti-science populism? That's really the only science politics I'm aware of now that creationism and climate skepticism have been firmly put to rest. > If only he had avoided the topic of Covid entirely, then he would be well regarded. I think it's more that his predictions were bad and poorly reasoned and he chose to defend them on right wind media outlets instead of making his case among scientists. He's not the first well-regarded scientist to go off on a politically-fueled side quest later in his career. Kary Mullis is a famous example.
    - kelipso 2y ago
      Ha ha, I suppose as long as he goes on left wing media outlets and make bad and poorly reasoned predictions that follow left wing politics, then he would be well regarded. And please, don't pretend like there is no left wing aligned science politics that is as much based on science as flat earthers. I assume you haven't been hibernating during the covid times. All the doctors who did exactly that are doing fine with regard to their reputations.
giantg2 2y ago
This must be a satire piece. It talks on things like power, reproducibility, etc. Which is fine. There are minority of papers with mathematical errors. What it fails to examine is what is "false". Their results may be valid for what they studied. Future studies may have new and different findings. You may have studies that seem to conflict with each other due to differences in definitions (eg what constitutes a "child", 12yo or 24yo?) or the nuance in perspective apllied to the policies they are investigating (eg aggregate vs adjusted gender wage gap). It's about how you use them - "Research suggests..." or "We recommend further studies of larger size", etc. It's a tautology that if you misapply them they will be false a majority of the time.
- austinjp 2y ago
  It's not satire. Ioannidis has a long history of pointing out flaws in scientific processes. (Edit: spelling.)
- dcl 2y ago
  What part about the genuine statistical arguments made in the article would make you believe it is satire? I've found the reaction to this article can be pretty intense. We read this in a journal club many years ago and one of the mathematicians who was kind of new to the idea that research papers (in other fields) didn't more or less represent 'truth' said this article was _dangerous_.
  - giantg2 2y ago
    It's ironic that the paper has a correction. The title and abstract sound highly editorial, especially given that "false" is never defined. They talk about pre study odds being an enhancement but I didn't see them include those in their own paper. The fact that a study is small or the impact is minor doesn't make the results false, especially when these limitations are called out and further research is requested. You could even have a case study n=1 be valid if the conclusion is properly defined. The main problem is people generalizing from things that don't have that level of support.
tombert 2y ago
As I’ve transitioned to more exploratory and researchy roles in my career, I have started to understand the science fraudsters like Jan Hendrik Schön. When you spent an entire week working on a test or experiment that you know should work, at least if you give it enough time, but it isn’t for whatever reason, it can be extremely tempting to invent the numbers that you think it should be, especially if your employer is pressuring you for a result. Now, obviously, reason we run these tests is precisely because we don’t actually know what the results will be, but that’s sometimes more obvious in hindsight. Obviously it’s wrong, and I haven’t done it, but I would be lying if I said that the thought hadn’t crossed my mind.
- bluefirebrand 2y ago
  > When you spent an entire week working on a test or experiment that you know should work I thought the whole point of doing experiments was to challenge what we "know" so we can refine our understanding?
  - MathMonkeyMan 2y ago
    Because of that, backing up a claim with research adds weight to the claim. If the claim is false, though, you can still sometimes get research to support it. If you or the researcher stands to profit from the false claim, then there is a conflict of interest.
  - llamaimperative 2y ago
    Sure in la-la-land where science isn't conducted by humans. In reality, scientists are highly motivated (i.e. biased) individuals like anyone else. Therefore science cannot be done effectively by individuals. The system that derives truth from experiments - the actual scientific system - is the competitive dynamic between scientists who are trying to tarnish each others' legacies and bolster their own. The scientific method etc. primarily makes scientific claims scrutinizable in detail, but without scrutiny they are still highly liable to produce false information.
    - 2y ago
DaoVeles 2y ago
It has been said that "Publish or Perish" would make a good tomb stone epitaph for a lot of modern sciences. I speak to a lot of people in various science fields and generally they are some of the heaviest drinkers I know simply because of the system they have been forced into. They want to do good but are railroaded into this nonsense for dear of losing their livelihood. Like those that are trying to progress our treatment of mental health but have ended up almost exclusively in the biochemicals space because that is where the money is even though that is not the only path. It is a real shame. Also other heavy drinkers are the ecologists and climatologists, for good reason. They can see the road ahead and it is bleak. They hope they are wrong.
angry_octet 2y ago
... including the junk pushed by Ioannidis. His completely trashed his credibility during COVID.
- pessimizer 2y ago
  By being less wrong than almost everyone else. Since everyone else was wrong together they shunned him (as science dictates), and now agree to not talk about how wrong they were.
  - angry_octet 2y ago
    He used his reputation and statistical expertise to mislead the world as to the true prevalence (COVID infection rate) and supported the fantasies of Bhattacharya, Kulldorff and Gupta. It is hard to estimate what effect his misinformation had on COVID control measures, and there was no shortage of attention seeking clowns, but he stepped up to the plate and he can take credit for some of the millions of deaths. It was scientific misconduct but his position shields him from consequences.
    - angry_octet 2y ago
      Your down votes are tears of COVID denialism.
23B1 2y ago
Imagine if tech billionaires, instead of building dickships and buying single-family homes, decided to truly invest in humanity by realigning incentives in science.
- joycesticks 2y ago
  Damn people are getting pretty good at manifesting these days Check out ResearchHub[1], it's a company founded by a tech billionaire that's trying to realign incentives in science [1] - https://www.researchhub.com/ https://www.researchhub.com/
  - 23B1 2y ago
    Heh, thanks.
  - dominikusbrian 2y ago
    [dead]
Animats 2y ago
How broad a range is this result supposed to cover? It seems to be mostly applicable to areas where data is too close to the noise threshold. Some phenomena are like that, and some are not. "If your experiment needs statistics, you ought to have done a better experiment" - Rutherford
SideQuark 2y ago
This paper, almost 20 years old, has plenty of follow-up work showing the claims in this original paper aren’t true. One simple angle is Ioannidis simply makes up some parameters to show things could be bad. Later empirical work measuring those parameters found Ioannidis off by orders of magnitude. One example https://arxiv.org/abs/1301.3718 https://arxiv.org/abs/1301.3718 There’s ample other published papers showing other holes in the claims. https://scholar.google.com/scholar?cites=15681017780418799273&as_sdt=800005&sciodt=0,15&hl=en https://scholar.google.com/scholar?cites=1568101778041879927... Google scholar papers citing this
- asgraham 2y ago
  re: the arxiv link Why is it that microarray true positive p-values follow a beta distribution? Following the citations led to a lot of empirical confirmation but I couldn't find any discussion of why. More to the point of this rebuttal, though: why would we expect the amalgamation of 70k micro-array experiments' abstract-reported p-values to follow a single beta distribution? And what about modeling the bias-induced bump of barely-significant results? If there's some theoretical reason why the meta-study can use the beta-uniform model, then I could see this being only a mild underestimation of the proportion of false positives (14%), but otherwise I'm confused how we can interpret this.
- mike_hearn 2y ago
  That arXiv paper seems to imply that a false positive rate of 14% is "a reliable record of scientific progress". Even if we take their claim at face value, 14% of medical papers making false claims seems way too high. I'm personally very skeptical it's that low. What I found during COVID is that entire literatures exist in this sort of weird space where you can't even say if they're true or false because the basic methodologies of the field don't even get you that far. Computational epidemiology never seemed to test its predictions against reality to begin with, so the whole concept of doing P-value analysis on such papers would be meaningless. Based on personal experience I'd feel like Ioannidis is directionally correct.
md224 2y ago
Something that continues to puzzle me: how do molecular biologists manage to come up with such mindbogglingly complex diagrams of metabolic pathways in the midst of a replication crisis? Is our understanding of biology just a giant house of cards or is there something about the topic that allows for more robust investigation?
meling 2y ago
I only read the abstract; “Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true.” True vs false seems like a very crude metric, no? Perhaps this paper’s research claim is also false.
deleted 2y ago
[deleted]
smeej 2y ago
This kind of report always raises the question for me of what the existing system's goals are. I think people assume that "new, reliable knowledge" is among the goals, but I don't see that the incentives align toward that goal, so I don't know that that's actually among them. Does the world really want/need such a system? (The answer seems obvious to me, but not above question.) If so, how could it be designed? What incentives would it need? What conflicting interests would need to be disincentivized? I think it's been pretty evident for a long time that the "peer-reviewed publications system" doesn't produce the results people think it should. I just don't hear anybody really thinking through the systems involved to try to invent one that would.
- sesm 2y ago
  My favorite contemporary physicist is doing a mundane job at NASA and does all the interesting theoretical research as a side project. I think this should be the default.
dang 2y ago
Related. Others? Why most published research findings are false (2005) - https://news.ycombinator.com/item?id=37520930 https://news.ycombinator.com/item?id=37520930 - Sept 2023 (2 comments) Why most published research findings are false (2005) - https://news.ycombinator.com/item?id=33265439 https://news.ycombinator.com/item?id=33265439 - Oct 2022 (80 comments) Why Most Published Research Findings Are False (2005) - https://news.ycombinator.com/item?id=18106679 https://news.ycombinator.com/item?id=18106679 - Sept 2018 (40 comments) Why Most Published Research Findings Are False - https://news.ycombinator.com/item?id=8340405 https://news.ycombinator.com/item?id=8340405 - Sept 2014 (2 comments) Why Most Published Research Findings Are False - https://news.ycombinator.com/item?id=1825007 https://news.ycombinator.com/item?id=1825007 - Oct 2010 (40 comments) Why Most Published Research Findings Are False (2005) - https://news.ycombinator.com/item?id=833879 https://news.ycombinator.com/item?id=833879 - Sept 2009 (2 comments)
- cb321 2y ago
  As clarification, the article linked in the subject is dated 2022 BUT it is actually just "a correction" of the very famous 2005 article. The correction is eentsy-weentsy - just a missing pair of parenthesis if you click through: There is an error in Table 2. A set of parentheses is missing in the equation for Research Finding = Yes and True Relationship = No. Please see the correct Table 2 here.
tdba 2y ago
One study tried to replicate 100 psychology studies and only 36% attained significance. https://osf.io/ezcuj/wiki/home/ https://osf.io/ezcuj/wiki/home/
ninetyninenine 2y ago
So whenever someone gives me a detailed argument with cited sources I can show them this and render the truth into an unobtainable objective.
hofo 2y ago
Oh the irony
dominikusbrian 2y ago
[dead]
iskander 2y ago
I think unpopular to mention here but John Ioannidis did a really weird turn in his career and published some atrociously non-rigorous Covid research that falls squarely in the cross-hairs of "why...research findings are false".
Daub 2y ago
From my experience, my main criticism of research in the field of computer vision is that most of it is 'meh'. In a university that focused on security research, I saw mountains of research into detection/recognition, yet most of it offered no more than slightly different ways of doing the same old thing. I also saw: a head of design school insisting that they and their spouse were credited on all student and staff movies, the same person insisting that massive amounts of school cash be spent promoting their solo exhibition that no one other than students attended, a chair of research who insisted they were given an authorship role on all published output in the school, labs being instituted and teaching hires brought in to support a senior admin's research interested (despite them not having any published output in this area), research ideas stolen from undergrad students and given to PhD students... I could go on all day. If anyone is interested in how things got like this, you might start with Margret Thatcher. It was she who was the first to insist that funding of universities be tied to research. Given the state of British research in those days it was a reasonable decision, but it produced a climate where quantity is valued over quality and true 'impact'.
- mike_hearn 2y ago
  Amazing, now I've really seen Thatcher blamed for everything. The problem of false scientific claims is global. The UK is by far not the worst offender (mostly places like Iran, India, Pakistan, Saudi Arabia, China are worst affected by paper mills to pick one problem...). And university funding has been tied to research everywhere for decades. It's really got nothing to do with Thatcher. Really, under an actually libertarian government university research wouldn't be funded at all. From their perspective companies are more than capable of doing research and the patent system encourages publication.
deleted 2y ago
[deleted]
DrNosferatu 2y ago
Have LLMs cross check papers and point out experiments to be repeated.
- pcrh 2y ago
  LLMs would not be very useful in this instance, since truly novel (and correct) findings would not have formed part of their training datasets.
fedeb95 2y ago
This published research is false. All published research will turn out to be false. The problem is ill-posed: can we establish once and for all that something is true? Almost all history had this ambition, yet every day we find that something we believed to be true wasn't. Data isn't encouraging.
- ImHereToVote 2y ago
  Maybe the real truth, was the friends we made along the way.
arrozconcosas 2y ago
Arguing why this paper is false is ironic in agreeing with the papers point
PeterZaitsev 2y ago
If most Published Research Findings are false, does not this mean this article is likely to be false as well ? :)
- dr-detroit 2y ago
  [dead]

[flagged]

JohnMakin 2y ago
That’s a lot of mental hoops you just leaped through. Are you saying you think immigration policy is the main contributing cause of poor research quality? Because that is a wild claim without evidence. Also doesn’t really make sense, immigration policy varies from state to state and there isn’t just one single country producing research.
- codingwagie 2y ago
  Some huge percentage of STEM graduate school is just for immigration These people are not seeking "truth"
  - lIllIlIIlIllIlI 2y ago
    https://en.wikipedia.org/wiki/No_true_Scotsman https://en.wikipedia.org/wiki/No_true_Scotsman
  - worik 2y ago
    Evidence? Especially given the context.
    - adamrezich 2y ago
      What kind of evidence would sway your belief on this matter? Perhaps a published academic research paper on the topic?
      - worik 2y ago
        > What kind of evidence would sway your belief on this matter? I do not have beliefs on this matter, I am interested in data. I would like to know what studies of graduate student intentions and outcomes > Perhaps a published academic research paper on the topic? I see what you (think you are) doing there. That would be a start
        adamrezich 2y ago
        It would be start, for people like you who hopelessly cling to your deference to the authority that academic research papers exert over people like yourselves. For the rest of us, it would be meaningless.
        worik 2y ago
        > For the rest of us, I am curious: What are your decision criteria? Academic research papers are systematically flawed, true. But they form a part of the story of finding truth. What do you use?
  - spicybright 2y ago
    Aren't they required to have previous schooling in their home country, pass an aptitude test against others, and maintain good grades to stay in the program in order to stay in the country? I don't see how you can avoid actually doing the education part here. And I'm sure lots see immigration + better education as a win/win, which I don't see a problem with. Where are you getting the huge percentage from? Do you have sources for your claim? Even news articles?
  - cortesoft 2y ago
    The numbers I find say about 20% of grad students in the United States are international students. I am not sure that is a huge percentage?
    - adamrezich 2y ago
      What does that statistic look like for other nations?
  - jjulius 2y ago
    >These people are not seeking "truth" Careful with those absolutes, there. And the whole "these people" thing, too, probably.
kimixa 2y ago
In 2022 there were ~57k PHDs awarded in the USA [0]. In the same year, there were ~500k immigrant visas [1]. If every single PHD was for immigration, it'll still only be ~10% of the total. And even if the thesis was "forced", just getting to PHD level means you're very much in the top echelons of education, and even then takes multiple years - as someone who immigrated with "only" a BSc I suspect they'll be able to use one of the many other paths instead with less effort. I'm not sure the logic holds. [0] https://www.forbes.com/sites/michaeltnietzel/2024/02/05/number-of-doctoral-degrees-awarded-in-us-rebounds-to-all-time-high/ https://www.forbes.com/sites/michaeltnietzel/2024/02/05/numb... [1] https://www.bal.com/perspectives/bal-news/united-states-us-visa-statistics-show-continued-increase-in-visa-issuance/ https://www.bal.com/perspectives/bal-news/united-states-us-v...

> In this framework, a research finding is less likely to be true [...] where there is greater flexibility in designs, definitions, outcomes, and analytical modes It's worth noting though that in many research fields, teasing out the correct hypotheses and all affecting factors are difficult. And, sometimes it takes quite a few studies before the right definitions are even found; definitions which are a prerequisite to make a useful hypothesis. Thus, one cannot ignore the usefulness of approximation in scientific experiments, not only to the truth, but to the right questions to ask. Not saying that all biases are inherent in the study of sciences, but the paper cited seems to take it for granted that a lot of science is still groping around in the dark, and to expect well-defined studies every time is simply unreasonable.

3np 2y ago
This is only meaningful if "the replicaton crisis" is systematically addressed.

marcosdumay 2y ago

Yeah, when you try new things, you often get them wrong. Why do we expect most published results to be true?

ekianjo 2y ago
because people believe that peer review improve things but in fact not really. its more of a stamping process
- elashri 2y ago
  Yes that a misconception that many people think that peer-review involves some sort of verification or replication which is not true. I would blame mainstream media in part for this and how they report on research and don't emphasize this nature. Mainstream media also is not interested in reporting on progress but likes catchy headlines/findings.
- deleted 2y ago
  [deleted]
bluefirebrand 2y ago
Because people use published results to justify all sorts of government policy, business activity, social programs, and such. If we cannot trust that results of research are true, then how can we justify using them to make any kind of decisions in society? "Believe the science", "Trust the experts" etc sort of falls flat if this stuff is all based on shaky research
- marcosdumay 2y ago
  > If we cannot trust that results of research are true, then how can we justify using them to make any kind of decisions in society? Well, don't. Make your decisions based on replicated results. Stop hyping single studies.
  - XorNot 2y ago

titanomachy 2y ago

2022

elashri 2y ago

There is at least one thing wrong about this. This is an essay about a paper published a simulation based scenarios in medical research. It then try to generalize to "research" and avoid this very narrow support to the claim. I think this is something true and it should make us more cautious when deciding based on single studies. But things are different in other fields. Also this is called research. You don't know the answer before head. You have limitations in tech and tools you use. You might miss something, didn't have access to more information that could change the outcome. That is why research is a process. Unfortunately common science books talks only about discoveries, results that are considered fact but usually don't do much about the history of how we got there. I would like to suggest a great book called "How experiments end"[1] and enjoy going into details on how scientific conscious is built for many experiments in different fields (mostly physics). [1] https://press.uchicago.edu/ucp/books/book/chicago/H/bo5969426.html https://press.uchicago.edu/ucp/books/book/chicago/H/bo596942...

thatguysaguy 2y ago
I think it's clear that this paper has stood the rest of time over the last 20 years. Our estimates of how much published work fails to replicate or is outright fraudulent have only increased since then.
- tptacek 2y ago
  Outright research fraud is probably very rare; the cases we've heard about stick out, but people outside of academia usually don't have a good intuition for just how vast the annual output of the sciences are. Remember the famous PhD comic, showing how your thesis is going to be an infinitesimal fraction of the work of your field.
  - dekhn 2y ago
    Is incompetence fraud? Or just incompetence? I'm asking because a fair number of the molecular biologists who get caught by Elizabeth Bik for copy/pasting images of gels insist they just made honest mistakes (with some commentary about the atrocious nature of record-keeping in modern biology). I alter Ionnides's conclusion to be instead: "Roughly 50% of papers in quantitative biological sciences contain at least one error serious enough to invalidate the conclusion" and "Roughly 75% of really interesting papers are missing at least one load-bearing method detail that reproducers must figure out on their own" (my own personal observations of the literature are consistent with these rates; I was always flabbergasted at people who just took Figure 3 as correct).

skybrian 2y ago

(2005). I wonder what's changed?

youainti 2y ago
Over on peerpub there has been some discussion of studies on the topic. https://pubpeer.com/publications/14B6D332F814462D2673B6E9EF9AF9#4 https://pubpeer.com/publications/14B6D332F814462D2673B6E9EF9...

motohagiography 2y ago

i wonder if science could benefit from publishing using pseudonyms the way software has. if it's any good, people will use it, the reputations will be made by the quality of contributions alone, it makes fraud expensive and mostly not worth it, etc.

wwweston 2y ago
People have uses for conclusions that sometimes don't have anything to do with their validity. So while "if it's any good, people will use it" is true and quality contributions will be useful, the converse is not true: the use or reach of published work may be only tenuously connected to whether it's good. Reputation signals like credentials and authority have their limits/noise, but bring some extra signal to the situation.
- motohagiography 2y ago
  what's missing from this paper is a probability using its own model that it too is false. counter to the headline, it implies that by its own probable falsehood, most published research is in fact true. I admit to missing the joke in the first reading. pseudonyms may prevent the abuse of invalid papers by removing the ability of the authors to front institutional reputations for partisan claims. the movement for science and data to drive policy outside of their domains sounds nice until you find that the science and data are irrepreducible, and the institutions have become laundering vehicles for debased opinions that wash the hands of policymakers. as though the potential for abuse has become the value. maybe it's a rarefied kind of funny, but the kernel of truth it reveals is that it could be time to start using pseudonyms in some disciplines to make the axis of policymakers and academics more honest.

debacle 2y ago

Most? Really?

ape4 2y ago

So is this paper false too? .. infinite recursion...

wccrawford 2y ago
Most probably.

youainti 2y ago

Please note the peerpub comments discussing that it appears that followup research shows about 15% is wrong, not the 5% anticipated. https://pubpeer.com/publications/14B6D332F814462D2673B6E9EF9AF9 https://pubpeer.com/publications/14B6D332F814462D2673B6E9EF9...

carabiner 2y ago

This only applies to life sciences, social sciences right? Or are most papers in computer science or mechanical engineering also false?

thatguysaguy 2y ago
It's very bad in CS as well. See e.g.: https://arxiv.org/abs/1807.03341 https://arxiv.org/abs/1807.03341 IIRC there was also a paper analyzing how often results in some NLP conference held up when a different random seed or hyperparameters were used. It was quite depressing.
pcrh 2y ago
It's mostly in medicine and psychology. In topics where there is less reliance on relatively small numbers of cases (as is typical for medicine), there is also less reliance on marginal, but statistically "significant", findings. So areas such as biochemistry, chemistry, even some animal studies, are less susceptible to over-interpretation or massaging of data.
mike_hearn 2y ago
Life sciences, social sciences, eco-sciences, humanities. Engineering is mostly OK or at least not mostly false. When you get into the humanities a lot of papers "aren't even wrong", as in, the authors don't hold themselves to any standards of logic or rigor to begin with, or aren't even making any kind of identifiable claim about the world, and don't consider rebuttals based on such criticisms to be valid. A lot of people overlook ecology/climatology because millenarians have made it into such a live wire, but those fields generally have worse problems than medicine. For instance, in medicine going back and retroactively changing patient records for your clinical trial is taboo. It happens remarkably often, but, everyone accepts that it's not supposed to. In climatology they retroactively change climate datasets all the time and if anyone calls them out on it they just attack the critics. They don't even accept the principle that their models should predict data as measured using a constant methodology. It's meaningless in such a field to say "the hypothesis is consistent with the data" because the historical data you analyzed might be replaced with a new version that's fundamentally different.

withinboredom 2y ago

I've implemented several things from computer science papers in my career now, mostly related to database stuff. They are mostly terribly wrong or show the exact OPPOSITE as to what they claim in the paper. It's so frustrating. Even occasionally, they offer their code used to write the paper and it is missing entire features they claim are integral for it to function properly; to the point that I wonder how they even came up with the results they came up with. My favorite example was a huge paper that was almost entirely mathematics-based. It wasn't until you implemented everything that you would realize it just didn't even make any sense. Then, when you read between the lines, you even saw their acknowledgement of that fact in the conclusion. Clever dude. Anyway, I have very little faith in academic papers; at least when it comes to computer science. Of all the things out there, it is just code. It isn't hard to write and verify what you purport (usually takes less than a week to write the code), so I have no idea what the peer reviews actually do. As a peer in the industry, I would reject so many papers by this point. And don't even get me started on when I send the (now professor) questions via email to see if I just implemented it wrong, or whatever, that just never fucking reply.

reasonableklout 2y ago
Wow, sounds awful. Help the rest of us out - what was the huge paper that didn't work or was actively misleading?
- withinboredom 2y ago
  I'd rather not, for obvious reasons. The less obvious reason is that I don't remember the title/author of the paper. It was back in 2016/17 when I was working on a temporal database project at work and was searching literature for temporal query syntax though.
Lerc 2y ago
For papers with code, I have a seen a tendency to consider the code, not the paper to be the ground truth. If the code works, then it doesn't matter what the paper says, the information is there. If the code doesn't work, it seems like a red flag. It's not an advantage that can be applied to biology or physics, but at least computer science catches a break here.

blackeyeblitzar 2y ago

It’s a matter of incentives. Everyone who wants a PhD has to publish and before that they need to produce findings that align with the values of their professors. These bad incentives combined with rampant statistical errors lead to bad findings. We need to stop putting “studies” on a pedestal.

breck 2y ago

On a livestream the other day, Stephan Wolfram said he stopped publishing through academic journals in the 1980's because he found it far more efficient to just put stuff online. (And his blog is incredible: https://writings.stephenwolfram.com/all-by-date/ https://writings.stephenwolfram.com/all-by-date/) A genius who figured it academic publishing had gone to shit decades ahead of everyone else. P.S. We built the future of academic publishing, and it's an order of magnitude better than anything else out there.

jordigh 2y ago
Genius? The one who came up with a new kind of science?
- breck 2y ago
  Do you think judging someone by your least favorite work of their's is a good strategy? Do you also say, "Newton a genius? The one who tried to turn lead into gold?"
  - jordigh 2y ago
    He still parades that around as his magnum opus in his latest blog post: https://writings.stephenwolfram.com/2024/08/five-most-productive-years-what-happened-and-whats-next/ https://writings.stephenwolfram.com/2024/08/five-most-produc... Yeah, I think it's fair to judge him by it.
    - breck 2y ago
      Fair enough. Like all of his work, I thought it was an incredible book, if you just randomly sample 10% of it. I never understood why he doesn't cut more, as he has genius ideas that get really watered down with lots of less relevant details. I would love if he started doing 1 page tldr's for all of his works.
wahern 2y ago
He created his own peer reviewed academic journal and founded a corporation to publish it: https://en.wikipedia.org/wiki/Complex_Systems_(journal) https://en.wikipedia.org/wiki/Complex_Systems_(journal) That's a little different than just putting stuff online.

ants_everywhere 2y ago

This is a classic and important paper in the field of metascience. There are other great papers predating this one, but this one is widely known. Unfortunately the author John Ioannidis turned out to be a Covid conspiracy theorist, which has significantly affected his reputation as an impartial seeker of truth in publication.

kelipso 2y ago
Ha how meta is this comment because the obvious inference one makes from the title is "Why Most Published Research Findings on Covid Are False" and that goes against the science politics. If only he had avoided the topic of Covid entirely, then he would be well regarded.
- ants_everywhere 2y ago
  It is pretty meta I guess. > Why Most Published Research Findings on Covid Are False Well, that's why there was so much focus on replication, multiple data sources and meta-analyses. The focus was there because the assumption is each study is flawed and those tools help extract better signal from the noise of individual studies. > and that goes against the science politics I don't think I follow you here. Are you referring to the anti-science populism? That's really the only science politics I'm aware of now that creationism and climate skepticism have been firmly put to rest. > If only he had avoided the topic of Covid entirely, then he would be well regarded. I think it's more that his predictions were bad and poorly reasoned and he chose to defend them on right wind media outlets instead of making his case among scientists. He's not the first well-regarded scientist to go off on a politically-fueled side quest later in his career. Kary Mullis is a famous example.
  - kelipso 2y ago
    Ha ha, I suppose as long as he goes on left wing media outlets and make bad and poorly reasoned predictions that follow left wing politics, then he would be well regarded. And please, don't pretend like there is no left wing aligned science politics that is as much based on science as flat earthers. I assume you haven't been hibernating during the covid times. All the doctors who did exactly that are doing fine with regard to their reputations.

giantg2 2y ago

This must be a satire piece. It talks on things like power, reproducibility, etc. Which is fine. There are minority of papers with mathematical errors. What it fails to examine is what is "false". Their results may be valid for what they studied. Future studies may have new and different findings. You may have studies that seem to conflict with each other due to differences in definitions (eg what constitutes a "child", 12yo or 24yo?) or the nuance in perspective apllied to the policies they are investigating (eg aggregate vs adjusted gender wage gap). It's about how you use them - "Research suggests..." or "We recommend further studies of larger size", etc. It's a tautology that if you misapply them they will be false a majority of the time.

austinjp 2y ago
It's not satire. Ioannidis has a long history of pointing out flaws in scientific processes. (Edit: spelling.)
dcl 2y ago
What part about the genuine statistical arguments made in the article would make you believe it is satire? I've found the reaction to this article can be pretty intense. We read this in a journal club many years ago and one of the mathematicians who was kind of new to the idea that research papers (in other fields) didn't more or less represent 'truth' said this article was _dangerous_.
- giantg2 2y ago
  It's ironic that the paper has a correction. The title and abstract sound highly editorial, especially given that "false" is never defined. They talk about pre study odds being an enhancement but I didn't see them include those in their own paper. The fact that a study is small or the impact is minor doesn't make the results false, especially when these limitations are called out and further research is requested. You could even have a case study n=1 be valid if the conclusion is properly defined. The main problem is people generalizing from things that don't have that level of support.

tombert 2y ago

As I’ve transitioned to more exploratory and researchy roles in my career, I have started to understand the science fraudsters like Jan Hendrik Schön. When you spent an entire week working on a test or experiment that you know should work, at least if you give it enough time, but it isn’t for whatever reason, it can be extremely tempting to invent the numbers that you think it should be, especially if your employer is pressuring you for a result. Now, obviously, reason we run these tests is precisely because we don’t actually know what the results will be, but that’s sometimes more obvious in hindsight. Obviously it’s wrong, and I haven’t done it, but I would be lying if I said that the thought hadn’t crossed my mind.

bluefirebrand 2y ago
> When you spent an entire week working on a test or experiment that you know should work I thought the whole point of doing experiments was to challenge what we "know" so we can refine our understanding?
- MathMonkeyMan 2y ago
  Because of that, backing up a claim with research adds weight to the claim. If the claim is false, though, you can still sometimes get research to support it. If you or the researcher stands to profit from the false claim, then there is a conflict of interest.
- llamaimperative 2y ago
  Sure in la-la-land where science isn't conducted by humans. In reality, scientists are highly motivated (i.e. biased) individuals like anyone else. Therefore science cannot be done effectively by individuals. The system that derives truth from experiments - the actual scientific system - is the competitive dynamic between scientists who are trying to tarnish each others' legacies and bolster their own. The scientific method etc. primarily makes scientific claims scrutinizable in detail, but without scrutiny they are still highly liable to produce false information.
  - 2y ago

DaoVeles 2y ago

It has been said that "Publish or Perish" would make a good tomb stone epitaph for a lot of modern sciences. I speak to a lot of people in various science fields and generally they are some of the heaviest drinkers I know simply because of the system they have been forced into. They want to do good but are railroaded into this nonsense for dear of losing their livelihood. Like those that are trying to progress our treatment of mental health but have ended up almost exclusively in the biochemicals space because that is where the money is even though that is not the only path. It is a real shame. Also other heavy drinkers are the ecologists and climatologists, for good reason. They can see the road ahead and it is bleak. They hope they are wrong.

angry_octet 2y ago

... including the junk pushed by Ioannidis. His completely trashed his credibility during COVID.

pessimizer 2y ago
By being less wrong than almost everyone else. Since everyone else was wrong together they shunned him (as science dictates), and now agree to not talk about how wrong they were.
- angry_octet 2y ago
  He used his reputation and statistical expertise to mislead the world as to the true prevalence (COVID infection rate) and supported the fantasies of Bhattacharya, Kulldorff and Gupta. It is hard to estimate what effect his misinformation had on COVID control measures, and there was no shortage of attention seeking clowns, but he stepped up to the plate and he can take credit for some of the millions of deaths. It was scientific misconduct but his position shields him from consequences.
  - angry_octet 2y ago
    Your down votes are tears of COVID denialism.

23B1 2y ago

Imagine if tech billionaires, instead of building dickships and buying single-family homes, decided to truly invest in humanity by realigning incentives in science.

joycesticks 2y ago
Damn people are getting pretty good at manifesting these days Check out ResearchHub[1], it's a company founded by a tech billionaire that's trying to realign incentives in science [1] - https://www.researchhub.com/ https://www.researchhub.com/
- 23B1 2y ago
  Heh, thanks.
- dominikusbrian 2y ago
  [dead]

Animats 2y ago

How broad a range is this result supposed to cover? It seems to be mostly applicable to areas where data is too close to the noise threshold. Some phenomena are like that, and some are not. "If your experiment needs statistics, you ought to have done a better experiment" - Rutherford

SideQuark 2y ago

This paper, almost 20 years old, has plenty of follow-up work showing the claims in this original paper aren’t true. One simple angle is Ioannidis simply makes up some parameters to show things could be bad. Later empirical work measuring those parameters found Ioannidis off by orders of magnitude. One example https://arxiv.org/abs/1301.3718 https://arxiv.org/abs/1301.3718 There’s ample other published papers showing other holes in the claims. https://scholar.google.com/scholar?cites=15681017780418799273&as_sdt=800005&sciodt=0,15&hl=en https://scholar.google.com/scholar?cites=1568101778041879927... Google scholar papers citing this

asgraham 2y ago
re: the arxiv link Why is it that microarray true positive p-values follow a beta distribution? Following the citations led to a lot of empirical confirmation but I couldn't find any discussion of why. More to the point of this rebuttal, though: why would we expect the amalgamation of 70k micro-array experiments' abstract-reported p-values to follow a single beta distribution? And what about modeling the bias-induced bump of barely-significant results? If there's some theoretical reason why the meta-study can use the beta-uniform model, then I could see this being only a mild underestimation of the proportion of false positives (14%), but otherwise I'm confused how we can interpret this.
mike_hearn 2y ago
That arXiv paper seems to imply that a false positive rate of 14% is "a reliable record of scientific progress". Even if we take their claim at face value, 14% of medical papers making false claims seems way too high. I'm personally very skeptical it's that low. What I found during COVID is that entire literatures exist in this sort of weird space where you can't even say if they're true or false because the basic methodologies of the field don't even get you that far. Computational epidemiology never seemed to test its predictions against reality to begin with, so the whole concept of doing P-value analysis on such papers would be meaningless. Based on personal experience I'd feel like Ioannidis is directionally correct.

md224 2y ago

Something that continues to puzzle me: how do molecular biologists manage to come up with such mindbogglingly complex diagrams of metabolic pathways in the midst of a replication crisis? Is our understanding of biology just a giant house of cards or is there something about the topic that allows for more robust investigation?

meling 2y ago

I only read the abstract; “Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true.” True vs false seems like a very crude metric, no? Perhaps this paper’s research claim is also false.

deleted 2y ago

[deleted]

smeej 2y ago

This kind of report always raises the question for me of what the existing system's goals are. I think people assume that "new, reliable knowledge" is among the goals, but I don't see that the incentives align toward that goal, so I don't know that that's actually among them. Does the world really want/need such a system? (The answer seems obvious to me, but not above question.) If so, how could it be designed? What incentives would it need? What conflicting interests would need to be disincentivized? I think it's been pretty evident for a long time that the "peer-reviewed publications system" doesn't produce the results people think it should. I just don't hear anybody really thinking through the systems involved to try to invent one that would.

sesm 2y ago
My favorite contemporary physicist is doing a mundane job at NASA and does all the interesting theoretical research as a side project. I think this should be the default.

dang 2y ago

Related. Others? Why most published research findings are false (2005) - https://news.ycombinator.com/item?id=37520930 https://news.ycombinator.com/item?id=37520930 - Sept 2023 (2 comments) Why most published research findings are false (2005) - https://news.ycombinator.com/item?id=33265439 https://news.ycombinator.com/item?id=33265439 - Oct 2022 (80 comments) Why Most Published Research Findings Are False (2005) - https://news.ycombinator.com/item?id=18106679 https://news.ycombinator.com/item?id=18106679 - Sept 2018 (40 comments) Why Most Published Research Findings Are False - https://news.ycombinator.com/item?id=8340405 https://news.ycombinator.com/item?id=8340405 - Sept 2014 (2 comments) Why Most Published Research Findings Are False - https://news.ycombinator.com/item?id=1825007 https://news.ycombinator.com/item?id=1825007 - Oct 2010 (40 comments) Why Most Published Research Findings Are False (2005) - https://news.ycombinator.com/item?id=833879 https://news.ycombinator.com/item?id=833879 - Sept 2009 (2 comments)

cb321 2y ago
As clarification, the article linked in the subject is dated 2022 BUT it is actually just "a correction" of the very famous 2005 article. The correction is eentsy-weentsy - just a missing pair of parenthesis if you click through: There is an error in Table 2. A set of parentheses is missing in the equation for Research Finding = Yes and True Relationship = No. Please see the correct Table 2 here.

tdba 2y ago

One study tried to replicate 100 psychology studies and only 36% attained significance. https://osf.io/ezcuj/wiki/home/ https://osf.io/ezcuj/wiki/home/

ninetyninenine 2y ago

So whenever someone gives me a detailed argument with cited sources I can show them this and render the truth into an unobtainable objective.

hofo 2y ago

Oh the irony

dominikusbrian 2y ago

[dead]

iskander 2y ago

I think unpopular to mention here but John Ioannidis did a really weird turn in his career and published some atrociously non-rigorous Covid research that falls squarely in the cross-hairs of "why...research findings are false".

Daub 2y ago

From my experience, my main criticism of research in the field of computer vision is that most of it is 'meh'. In a university that focused on security research, I saw mountains of research into detection/recognition, yet most of it offered no more than slightly different ways of doing the same old thing. I also saw: a head of design school insisting that they and their spouse were credited on all student and staff movies, the same person insisting that massive amounts of school cash be spent promoting their solo exhibition that no one other than students attended, a chair of research who insisted they were given an authorship role on all published output in the school, labs being instituted and teaching hires brought in to support a senior admin's research interested (despite them not having any published output in this area), research ideas stolen from undergrad students and given to PhD students... I could go on all day. If anyone is interested in how things got like this, you might start with Margret Thatcher. It was she who was the first to insist that funding of universities be tied to research. Given the state of British research in those days it was a reasonable decision, but it produced a climate where quantity is valued over quality and true 'impact'.

mike_hearn 2y ago
Amazing, now I've really seen Thatcher blamed for everything. The problem of false scientific claims is global. The UK is by far not the worst offender (mostly places like Iran, India, Pakistan, Saudi Arabia, China are worst affected by paper mills to pick one problem...). And university funding has been tied to research everywhere for decades. It's really got nothing to do with Thatcher. Really, under an actually libertarian government university research wouldn't be funded at all. From their perspective companies are more than capable of doing research and the patent system encourages publication.

deleted 2y ago

[deleted]

DrNosferatu 2y ago

Have LLMs cross check papers and point out experiments to be repeated.

pcrh 2y ago
LLMs would not be very useful in this instance, since truly novel (and correct) findings would not have formed part of their training datasets.

fedeb95 2y ago

This published research is false. All published research will turn out to be false. The problem is ill-posed: can we establish once and for all that something is true? Almost all history had this ambition, yet every day we find that something we believed to be true wasn't. Data isn't encouraging.

ImHereToVote 2y ago
Maybe the real truth, was the friends we made along the way.

arrozconcosas 2y ago

Arguing why this paper is false is ironic in agreeing with the papers point

PeterZaitsev 2y ago

If most Published Research Findings are false, does not this mean this article is likely to be false as well ? :)

dr-detroit 2y ago
[dead]