12 ms·

Welcome to science folks. Having worked a decade around a campus regularly sitting in on presentations (and being in research myself) I can say I believe 80% of

• api 4y ago
  You get what you incentivize, and this is what we incentivize. It’s probably about the worst in any field dominated by studies and statistics. Harder sciences are harder to fake as bad results can be more conclusively falsified.
  • melling 4y ago
    Yes, it’s great when we rediscover some nugget of wisdom that everyone has already explained. It’s quite useful to listen to Buffet and Munger. They’ve accumulated lots of wisdom. “You get what you reward for” https://youtu.be/hJYLJRr3hEY https://youtu.be/hJYLJRr3hEY This hour of Munger is well worth it: https://youtu.be/4ICaAKuAudQ https://youtu.be/4ICaAKuAudQ Here’s the transcript: https://fs.blog/great-talks/psychology-human-misjudgment/ https://fs.blog/great-talks/psychology-human-misjudgment/
    • lettergram 4y ago
      I think any parent or manager also knows this…
  • fweimer 4y ago
    Even in the hard sciences, the results are fundamentally about statistics. And they often require very expensive machines for reproduction. I'm not sure if it is a good idea to try to optimize the scientific process for fraud resilience. In the end, we still rely on people doing the right thing most of the time.
• nonrandomstring 4y ago
  > (1) As a researcher you lose money if you don’t produce This is the biggest problem in my view. My work in R&D taught me that most of the time we don't produce anything. It's high risk. But it's high in rewards, often in adjacent areas not primarily the focus of the initial brief. Surely all serious investors understand this. Research is something we do for marginal returns. It's not an "innovation factory". With things currently stacked against risk, research can only yield tepid results.
  • throwawaylinux 4y ago
    > Surely all serious investors understand this. Not necessarily when the investor is an academic institution or a government.
    • nonrandomstring 4y ago
      By "investor" I was thinking of people who risk their own money.
      • throwawaylinux 4y ago
        One of the major root causes of the problems with the system.
  • ncmncm 4y ago
    Investors are making solar farms be sited in deserts, are pouring money into fusion startups, and bid up fraudulent Energy Vault (NRGV) to $2B. Investors canonically do not understand.
• zeristor 4y ago
  The thing is the difference between “Science” provable hypothesis, challenges and the current research industry. It would appear that this isn’t really “Science” if the ideal of research is being veered away from so much. Vested interests suffocating the process.
• redeeman 4y ago
  would be nice if people kept this in mind before they crucify others for not "trusting the science"
  • itsdifferent 4y ago
    • cersa8 4y ago
      It would be insane to not question the narrative given the incentives.
    • redeeman 4y ago
      agreed. it is however quite easy to ignore one side, quite a bit harder to ignore another that wields extreme power.
• hammock 4y ago
  • riley_dog 4y ago
    I saw this comment coming a mile away...
    • nathanaldensr 4y ago
      Good for you. Maybe you should ask yourself why so many people are skeptical? Is it really that they are all crazy wingnuts, or is it perhaps that humans have shown for all of our existence to be untrustworthy, greedy, power-hungry creatures?
  • raverbashing 4y ago
    Because 90% of it was not reasonable doubt, it was manipulative BS (If you can discuss the fine points between "lab leak theory" and "result of genetic manipulation" without mentioning any politician or anything that came out of a fb page or entertainment channel with "news" in its name that's a good discussion to have
    • mandmandam 4y ago
      ... To balance your point, which tbf has some truth in it, 99% of coverage on that topic from so called respected sources was manipulative bullshit. Dissenting opinions were censored by the millions across social media. Meanwhile, anyone who felt the lab leak theory was at least worth investigating was accused of being a Russian, Trumpist, anti-science piece of shit. Daszak's paper was obvious bullshit. Not in hindsight, not on close reading, but on immediate reading by anyone with basic experience and logic. Yet across government, media and academia it was cited as the definitive scientific position on the matter, case closed, stfu. ... So if you're going to blame people for mixing their discussion with corporate news and political stunts, then you're going to have to share that blame with all responsible parties. They've fully earned the ire and distrust which many across the political spectrum now hold them in. And we still haven't had a proper investigation in Wuhan, there's been no accountability for Daszak's conflict of interest, there's been no one held to account for the unprecedented censorship across Twitter and FB. No corporate media have been held responsible for their bullying and their bullshit. If you want people to have smarter discussions about this, I'd remember the above before accusing everyone of just 'getting their info from politicians and FB'.
• ad404b8a372f2b9 4y ago
  I don't have much to add, only that it's been my personal experience as well. Fraud and non-reproducibility used to be such ethereal concepts to me, issues that are discussed at a systemic level as an intellectual exercise. But after becoming a scientist, coming across so many fraudulent papers, so much non-reproducible and poorly done research. And seeing how it's affected my own, the thing that's dearest to me, having to build on top of those results and work in that global environment. It's been heart breaking, I feel no love for science anymore.
  • epistasis 4y ago
    What field are you in where you are coming across lots of fraudulent papers?! I spend most of my career reanalyzing others' data and combining it together into larger datasets, in molecular biology and genomics. The only time I encountered mistakes it was from my own labeling errors or bugs, pre-publication. And in pre-publications datasets I would sometimes detect accidental swaps on the labels of samples as part of QC checks.
    • ad404b8a372f2b9 4y ago
      Machine learning
      • riffraff 4y ago
        But surely ML did advance a lot in recent years? As the saying goes, science advances one funeral at a time, but it does advance. It seems overall you can still have an optimistic point of view of science while accepting it does not work as well as it could.
        • ad404b8a372f2b9 4y ago
          That's fair but that's the systemic view of the issue. When you are in the trenches you can't look at the good and throw away the bad. You have to compare your work to the fraudulent work, spend a lot of time reading and trying to reproduce the fraudulent work, cite the fraudulent work, compete with the fraudulent work, publish as fast they do, with results better than they produce. And even the work that's not fraudulent but just bad or rushed. It's a race to the bottom, and how can we ask that people compromise the quality of their work to compete when the only motivation for doing this work is the love of the craft. It's certainly not the money, or the environment, or the work-life balance, or the prestige.
• FredPret 4y ago
  We should introduce a maximum number of papers per researcher per period, like one per year or one per quarter. That way the only possible competition will be number of citations. The pressure will be for quality, not quantity.
  • Turing_Machine 4y ago
    > The pressure will be for quality, not quantity. Well, popularity, anyway. While popularity is sometimes a useful proxy for quality, that's not always the case.
    • FredPret 4y ago
      Probably the best proxy for quality is popularity that endures for centuries since we can’t measure quality directly. Either way, if there’s a max of one paper per year, scientists will be trying for the most impactful work.
      • Retric 4y ago
        People are already boosting their impact scores by requiring papers they review to add citations for their past work. Performance Metrics always result in people gaming the system, and impact scores have quickly become unreliable.
        • ModernMech 4y ago
          At the same time, reviewers are chosen based on their expertise in a field, and one of the things reviewers look for is that new papers cite relevant prior research. Are you suggesting that reviewers should just let papers go through without citing relevant prior research if it is their own research? Does it actually matter whose research it is as long as it is relevant prior research?
          • Retric 4y ago
            Of course it matters, adding citations to already written research is deceptive and thus actively harmful. The point of citations isn’t simply an index of prior research it’s there to inform readers of the writers context.
• Calavar 4y ago
  I agree with everything on your list. Most of these problems are quite difficult to solve, but I think (2) is doable if we can convince politicians to unfreeze science spending that has been stagnant for decades. We need a new set of early career awards that require 50% of time/effort to be spent on reproduction studies.
  • ModernMech 4y ago
    > We need a new set of early career awards that require 50% of time/effort to be spent on reproduction studies. Then you also need a realignment of the entire academic system as well. A new professor spending 50% of their time redoing the experiments of others, especially on capital expenditures and grad student time is going to be at a severe disadvantage against others who spent 100% of their time building their own research agenda. We'd probably need a whole new degree that's like a PhD, but whose dissertation isn't a novel extension of the field, but reproduction of other studies. Because as it is, a grad student can spend up to a decade on a Ph.D. just doing novel research. Now you want them to spend 50% of their time on reproduction? So is a Ph.D. supposed to take up to 20 years with your proposal? Or does a dissertation contain 50% less novel research? Certainly we should have funding for reproduction studies, but it's not enough to just have a grant; you have to have a valid career path doing such work. That doesn't exist today.
    • Calavar 4y ago
      > We'd probably need a whole new degree that's like a PhD, but whose dissertation isn't a novel extension of the field, but reproduction of other studies. I don't think that would work - those positions would go empty. People apply for grad school because they want to do original research. On the other hand, once people are already in the academic pipeline and they run into struggles advancing from postdoc to assistant prof or maintaining funding as an assistant prof, I think a lot of people would take a 50/50 grant if lets them keep one foot in the door for original research. They aren't going to turn their noses up at the only grants they are competitive for. You do raise a good point that labs with 50/50 grants wouldn't have as good of a value proposition for prospective grad students as labs doing 100% original research. I'm not sure what the solution is there.
• endgame 4y ago
  https://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/ https://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-in... Humans are not incentive-following automata, though we have incentive-following tendencies.
• pcurve 4y ago
  I have a friend with degree and Ivy, got phd, and did post doc. He said something exactly the same, and his conscious wouldn't allow him to work in the field of science. He left the field many years ago. He delivers mails now.
  • themitigating 4y ago
    What is the field of science
    • calvinmorrison 4y ago
      similar to the field of dreams but motivated by money instead of altruism
      • themitigating 4y ago
        So capitalism?
• codethief 4y ago
  > I believe 80% of science wrong AND not reproducible Maybe that's the case in your area of research. In mine (math, physics) it definitely isn't. So I would be a bit more careful about the wording here. ("80% of science" – what science?)
  • OrangeMonkey 4y ago
    In healthcare and life sciences research, I absolutely believe at least 80% is wrong and/or not reproducible. Not due to bad actors but, more likely, dataset bias or accidents upon patient matching algorithms or confounders. P < 0.05 is outdated at this point, for anything truly ground breaking p < 0.005 or < 0.0005 is probably a better choice, and even then I would ask "Where did you get your dataset from and did you combine (!!!) datasets from multiple orgs."
    • nradov 4y ago
      Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. https://doi.org/10.1371/journal.pmed.0020124 https://doi.org/10.1371/journal.pmed.0020124
      • epistasis 4y ago
        This paper would be more relevant if most science came down to evaluating a single p-value. Perhaps there are some fields of science where that is true, but I have not encountered them. I mostly look at molecular biology and infectious disease. Clinical trials in medicine come down to a single p-value by design, but don't fall prey to what he's talking about.
    • themitigating 4y ago
      What is the source for your belief? Something unscientific perhaps?
      • OrangeMonkey 4y ago
        The source for my beliefs is that I perform life sciences research to either generate new studies that are for publication or validate our studies against studies ran in the past. Its actually my job.
• rgrieselhuber 4y ago
  Wait, doesn’t this mean that funding sources can impact what “science” gets settled?
  • nathanaldensr 4y ago
    A more cynical take (or maybe realistic?) is that science becomes "settled" once the revenue streams are established.
    • rgrieselhuber 4y ago
      Which would also imply a total lack of funding for counter-narrative research.
• OrangeMonkey 4y ago
  100% agreed. I can't seem to get people to understand - most research studies find results that are not reproducible. Putting it nicely "Research has a reproducibility problem."
• synu 4y ago
  80% of science? How would things like GPS or rockets work if that was the case? What we consider reliable technology would be blowing up everywhere around us constantly.
  • woevdbz 4y ago
    Science, not engineering
    • synu 4y ago
      Is rocket science possible sans science? Can you be more specific what engineering without dependency on any science is? Like how do you even approach inventing gps as an engineer if 80% of physics calculations don’t actually work for example?
      • ncmncm 4y ago
        Almost all of what is called "rocket science" is firmly in the engineering wheelhouse. GPS relies very little on 20th century science, beyond GR corrections. The science behind orbital mechanics is just Newton's Laws, stuff thoroughly settled centuries ago.
    • orzig 4y ago
      Where do you think engineers get their equations?
      • hcks 4y ago
        For the science part of it: >100 years old scientific literature
        • synu 4y ago
          Possibly this was sarcasm, but I’ll bite just in case. There is quite a lot of scientific research that informs modern aerospace engineering: https://phys.org/tags/aerospace+engineering/ https://phys.org/tags/aerospace+engineering/ You’ll find similar physics research for example in nuclear power, optics, quantum computers, telecommunications, engines.. all kinds of engineering really. Chemistry is also quite relevant and there are new, stronger materials being engineered all the time based on research. It’s definitely not all just 1900s physics driving everything, even if you say that for some reason anything researched then doesn’t fall under the 80% of science that doesn’t work (which GP didn’t).
• themitigating 4y ago
  First off science isn't a section of society, company, or group of people. Second 80%? Your wild hyperbole has no value, makes no point, and reeks of immaturity
• Gimpei 4y ago
  I dunno, I feel like it’s bad but getting better. 20 years ago, nobody tried to replicate anything. Now it’s much more common and, I hope, expected for results like this one.
• dahart 4y ago
  > Welcome to science folks. There is no question that a very significant portion is wrong and some of it is fraud, but I feel like this is the wrong summary takeaway. Science isn’t the reason that science is messed up, people are the reason. All fields of human endeavor suffer from the same problems due to emotional and political and selfish people. It’s more like, welcome to humanity. Science is actually the best thing we’ve got, there is no alternative that has less BS and more truth. The 10% or so of science that’s right has transformed the earth in the last century. I worry about framing this the right way, about the subtleties of how you say it, because there is currently a war being waged on public trust in science, and to some degree that anti-science war is being won. It’s potentially damaging to say “most of science is wrong” and just stop there. That’s a misleading framing in my opinion. In order to fix the funding problems, society as a whole needs to have trust in science, to believe that the majority of people doing science are politically impartial and also not wasting money or lining their own pockets, to believe that scientific progress is human progress. It’s important to note that the incentive problems you cite mostly aren’t caused my any malicious intent. Disagreeing with someone else’s research, in my experience, isn’t often done with the primary goal of holding back good research, it’s done because the researchers actually disagree on the science, and the reviewer actually believes the proposed paper isn’t complete or correct or up to publication standards. Moreover, for science it’s very important for researchers to be critical of each other. That is part of why we need more replication study.
  • charlieyu1 4y ago
    The current funding system needs a massive change at least. Funding should go to universities in general instead of specific projects. Get rid of most of the KPIs.
    • dahart 4y ago
      I’ve never seen KPIs in academics, is that a thing now? I’ve seen KPIs in tech companies who like to model Google. There is a lot of funding going to universities, and a lot going to specific projects. I think maybe the main model for research funding generally is that funding doesn’t just appear, people have to go ask for it in the form of a grant proposal. Those grants typically once granted go 40% to the university anyway. Universities, for their part, have been absorbing money for larger and more expensive administrations for the last 30-40 years, and it’s out of hand. Most people I know practicing science feel the opposite of what you said - that less funding should go to the university and more directly toward funding research.
• throwawaymaths 4y ago
  There's probably a higher percentage if the article comes from the journal organic syntheses
• orzig 4y ago
  For everyone reading, and appropriately outraged, I challenge you to put your money where your mouth is. Even $5 to someone with a long history of finding scientific fraud puts you on the right side of history: https://www.patreon.com/elisabethbik https://www.patreon.com/elisabethbik I’ve been donating for years and I’ve never been more proud of that decision.
• hownottowrite 4y ago
  Yes indeed. The sexy world of academia. Fast cars and easy money.
• ta988 4y ago
  In biology and some parts of chemistry, I would say yes 80% of papers have a wrong conclusion (experiments are ok but not interpreted correctly) or completely wrong. But in other fields it is not the case. I'm pretty sure mathematics are much better. Physics it will depend on which branch some are closer to science-fiction anyway so are not verifiable, but most physics is good quality. It is hard to consider science as a whole because it is absolutely not an homogenous lump and practices in reviewing, publishing and peer pressure are radically different.
• deleted 4y ago
  [deleted]
• stewbrew 4y ago
  Ad 5: more data doesn't help if the main problem of your research is that your data is shit. People benefit from generating tons of shity data because they get more money, which looks good on their resume. If their results are shit, they simply write "more data is needed" and they benefit even more from the call for more data.