41 ms·

Time to assume that health research is fraudulent until proven otherwise?

• peytn 5y ago
  No, but you need to be able to ask people who’re “in the know,” e.g. > When Roberts contacted one of the journals the editor responded that “I wouldn’t trust the data.” People already know what the bullshit results are.
  • KittenInABox 5y ago
    How do you know if any individual person is "in the know"?
    • wrycoder 5y ago
      You evaluate their background, publications, and associates, and then you use your best judgement.
    • earleybird 5y ago
      With an appeal to authority :-)
  • caseysoftware 5y ago
    If the editor didn't trust the data, why did they publish? The people who keep informed on their field (aka "in the know") would then be tainted because they would likely believe a journal would vet the data, process, and researchers before publication. Unless you mean "in the know" in that they know the entire publishing system is a scam... and the ripples from that are huge.
    • topspin 5y ago
      "If the editor didn't trust the data, why did they publish?" They're in on it. Without material the journals have nothing to publish. So their inclination is to accept and publish with as little friction as possible. Publish or Die. Remember? That applies to the whole supply chain, not just poor put upon individual researchers.
  • diognesofsinope 5y ago
    Similar to how in psychology replication studies when they asked professors to predict which studies would and wouldn't replicate I think they were ~75% correct. Replication was ~50%. They have an idea of what is bullshit, but there's a very strong culture of 'don't call others out on bad research'
• gotoeleven 5y ago
  The war on standards is creating a low trust society.
  • onlyrealcuzzo 5y ago
    I don't know. The amount of people that take one poorly measured data point and use that as a north star seems more troubling to me - than a society with low trust. I'm not convinced a low-trust society is inherently bad. I am convinced that a society that has high trust in garbage data is bad.
    • long_time_gone 5y ago
      ==I'm not convinced a low-trust society is inherently bad. I am convinced that a society that has high trust in garbage data is bad. == It feels like we currently have both at the same time. There is "low trust" in any data that doesn't confirm an existing bias, but "high trust" in the data we want to be true. I see it all the time in HN comments. Any study that contradicts the consensus is met with comments about "correlation =/= causation", while anything that supports the census is supported and correlation/causation isn't mentioned.
      • UnFleshedOne 5y ago
        I think a better term for that is "brain damage". So what if all our brains are built damaged. Saying people are not reasoning machines, but social maneuvering machines doesn't change the fact they are kinda broken...
        • bjornsing 5y ago
          The thing is, I think people can be both reasoning machines and social maneuvering machines, depending on the incentive structure. Unfortunately we’ve created a society that greatly incentives social maneuvering.
    • patrec 5y ago
      I know this is a forum for nerds, but I struggle to understand how anyone could seriously believe that being surrounded by people who are either psychopathic or paranoid is probably A-OK, as long as they are all competent statisticians.
• 0xbadc0de5 5y ago
  Wouldn't this be the default scientific position? Or to take some of the hyperbole out of the statement, rephrase as: "is it time to assume that health research hypotheses are incorrect until a preponderance of data and reproducible studies prove otherwise?"
  • Dylan16807 5y ago
    "preponderance" is doing a lot of work in that statement. I would argue that this is mostly about reevaluating our standards for preponderance. And it's really hard to know if a study is reproducible. We could assume everything is incorrect until it has already been reproduced by an unrelated party, and I think that would be a major change in thought.
  • ChuckMcM 5y ago
    Exactly, that's the thing about science, nobody believes it until everything that has been tried to show it is wrong fails. I wish this were taught more in schools.
    • hanniabu 5y ago
      Yeah but you also can't go test 100 previous studies that the work you want to do is based on before you can even start yours. That's extremely inefficient, wasteful, and will tremendously slow progress.
      • guscost 5y ago
        > That's extremely inefficient, wasteful, and will tremendously slow progress. Democracy is also “extremely inefficient, wasteful, and slow to progress”. A process with those deficiencies can still be the best available approach (although your example is perhaps too far in that direction).
  • A4ET8a8uTh0 5y ago
    Scientific.. yes. But the studies are intended and/or used for a specific business purpose. The moment you recognize this simple reality, it becomes extremely difficult to take anything at face value. My wife is on the other side of the spectrum. She explicitly believes that companies/researchers/people generally want to do the right thing. It is infuriating, because my personal approach is my approach to games: 'shit, until proven otherwise'.
• rohanphadte 5y ago
  Some highlights to show how health research is published: > Mol, like Roberts, has conducted systematic reviews only to realise that most of the trials included either were zombie trials that were fatally flawed or were untrustworthy. > But the anaesthetist John Carlisle analysed 526 trials submitted to Anaesthesia and found that 73 (14%) had false data, and 43 (8%) he categorised as zombie. When he was able to examine individual patient data in 153 studies, 67 (44%) had untrustworthy data and 40 (26%) were zombie trials. > Others have found similar results, and Mol’s best guess is that about 20% of trials are false. Very few of these papers are retracted.
  • mcguire 5y ago
    You missed a sentence. "Many of the trials came from the same countries (Egypt, China, India, Iran, Japan, South Korea, and Turkey), and when John Ioannidis, a professor at Stanford University, examined individual patient data from trials submitted from those countries to Anaesthesia during a year he found that many were false: 100% (7/7) in Egypt; 75% (3/ 4) in Iran; 54% (7/13) in India; 46% (22/48) in China; 40% (2/5) in Turkey; 25% (5/20) in South Korea; and 18% (2/11) in Japan." I find it particularly sad, since actively promoting academic integrity would do more for those countries than anything else, bang-for-your-buck-wise. Instead, many seem to be seeking the appearance of academic success. (OTOH, I suppose Japan and South Korea may be on that list due to some kind intense pressure to succeed.)
    • bulletsvshumans 5y ago
      What about specifically applying increased scrutiny to institutions/countries/other identifiable cohorts? Is it too impolitic?
• codingwageslave 5y ago
  Non elite stem academia is essentially a back door immigration program. The research is basically useless, and incremental at best
• satchlj 5y ago
  This should be the default scientific position, however because people (including scientists) care greatly about their health and the health of their loved ones, they are very likely to latch on to things that they would like to be true science is almost never practiced in its ideal form; maybe it's Time to assume that results from our scientific institutions are flawed which is my assumption
• east2west 5y ago
  Already reached the conclusion when I saw a genetic researcher presenting his p-value < 10^-40 as better than < 10^-10. I kept my mouth shut because I didn't want to ruin the poor guy's moment in the sun, but I knew it was time to get out.
  • function_seven 5y ago
    My naive understanding is that "smaller p-value" == "more likely result is true". I know there's always more nuance in statistical reasoning, but the first number is vastly smaller than the second one, right? Is it just that both are hilariously tiny and not credible? Or is there no additional value after you get into the one-in-billions territory?
    • native_samples 5y ago
      Yeah I'm also mystified by that comment. You are correct that smaller P is better. Those near physics level P-values are not totally unheard of for genetics either, because they have very large databanks with hundreds of thousands of data points in them and the ability to do large analyses over them, so they can obtain a lot of statistical power.
      • CrazyStat 5y ago
        Precision in p-values that small is more or less meaningless in almost all cases, because any violation of model assumptions will result in p-value imprecision far greater than 10^-10. p-values are (almost always) approximations based on an approximate model, and the variation between the model and reality is probably more than 10^-10. Some tiny aspect of the real process that your model falls to capture might mean that that 10^-10 is actually 0.001, and 10^-40 is also 0.001. In complex biological fields it's fair to assume that there are always such tiny aspects.
        • native_samples 5y ago
          You're right. The numbers are too small to be plausible. I read on Scott Alexander's blog about 5-HTTLPR that in genetics they can get very low P values relative to most life sciences, but 10^-40 indeed seems far too low for any plausible experiment. I guess even in particle physics they don't go that low.
• MauranKilom 5y ago
  This article talks about "zombie trials" as if it were known to the reader what exactly that means. Anyone who could clue me in?
  • danuker 5y ago
    Following the reference trail leads to this: > I categorised trials with false data as ‘zombie’ if I thought that the trial was fatally flawed. https://associationofanaesthetists-publications.onlinelibrary.wiley.com/doi/10.1111/anae.15263 https://associationofanaesthetists-publications.onlinelibrar...
  • nabla9 5y ago
    >I have chosen the word ‘zombie’ to indicate trials where false data were sufficient that I think the trial would have been retracted had the flaws been discovered after publication. The varied reasons for declaring data as false precluded a single threshold for declaring the falsification sufficient to deserve the name ‘zombie’, 1. Carlisle JB. False individual patient data and zombie randomised controlled trials submitted to Anaesthesia. Anaesthesia 2020; https://doi.org/10.1111/anae.15263 https://doi.org/10.1111/anae.15263.
• argvargc 5y ago
  Yes, though it's not much of a problem for those unwilling to take the leap based on only one or two emergent studies, who might prefer relying on decades of safety data. Oh, but then that entirely sensible and understandable action might get one censored, deleted, fired, fined and/or discriminated against as a third class citizen. Cui bono?
  • tasogare 5y ago
    Exactly. I've argued on social media with "trust the science" people: none of them were scientists while I am one. People in general have no idea how the sausage is made and how broken the system is. A year or more ago I was joking with a coworker "I hope medical studies are done more seriously than what we do". Today I wouldn't make the same joke given how the situation went out of control with states enforcing authoritarian policies based on poor understanding of how science work.
• adolph 5y ago
  Is this some vaccine hesitancy in disguise from BMJ? Stephen Lock, my predecessor as editor of The BMJ, became worried about research fraud in the 1980s, but people thought his concerns eccentric. Research authorities insisted that fraud was rare, didn’t matter because science was self-correcting, and that no patients had suffered because of scientific fraud. All those reasons for not taking research fraud seriously have proved to be false, and, 40 years on from Lock’s concerns, we are realising that the problem is huge, the system encourages fraud, and we have no adequate way to respond.
  • zucker42 5y ago
    What? How do you figure that? Some person points out actual instances of poor quality research and you imply he is anti-vaccine? I can find no record of this person discouraging vaccination.
    • adolph 5y ago
      Ok, maybe its just a coincidence that it was published right when the current dangers of misinformation are so high and so many are failing to believe science.
      • wizzwizz4 5y ago
        The solution to “people aren't believing academia” is to make academia more trustworthy (probably by filtering for trustworthiness rather than making individuals more trustworthy), not to encourage scientism. Seeking the truth is still important, after all.
        • adolph 5y ago
          Wasn’t trustworthiness what “peer reviewed journals” was all about? Clearly research fraud isn’t “seeking the truth.”
      • zucker42 5y ago
        Well of course I'm not going to believe blatantly fraudulent "science". That has nothing to do whether or not I would take a COVID vaccine (I have). Trying to suppress the truth that there is shoddy or false research to keep up appearances is ridiculous. There's something between ignoring scientific evidence you dislike and credulously taking everything a scientist says as fact. And the "dangers of misinformation" have been high for a long time now (perhaps since antiquity). Modern climate change denial has its roots in the 90s, and modern vaccine skepticism is similarly old.
• injidup 5y ago
  I recently had the privilege of trying to do the right thing when I identified fraudulent research carried out by an institution in Austria. The initial response of the institution was positive but when I pressed for further details on how such things could happen suddenly nobody anymore would take my call. The research was paid for by a private company to pimp a nonsense product. The research was never published in a research journal but it didn't stop the company using the name of the university alonside exerpts from the paper in marketing material alongside gushing claims of "proved by science". The company threatened to sue me and the university threatened me as well. Neither has followed through on threats. The company wants to keep selling their rubbish magnetic health ding ding and I assume the university wants nobody to look into how positive results for the product came out of their institution. Allround an education on how the real world works.
  • birdyrooster 5y ago
    I guess if you don't even feel comfortable sharing the identity of any of the parties, we really have no chance to identify bad actors from good as a community.
    • injidup 5y ago
      https://shop.powerinsole.com/en/blogs/news/doppelblindstudie-der-universitat-salzburg https://shop.powerinsole.com/en/blogs/news/doppelblindstudie... I actually bought one of the devices and took it apart on video. There is nothing inside except 4 magnets and a plastic printed card. No active circuits and no power source and no components such as resistors or capacitors and certainly there is no microchip inside. edit below are links to pictures of the device and accompanying text as originally on facebook posted ======================================= Hi Powerinsole I ordered one of your power chip devices and took a detailed look at it. My analyses is as follows. 1. There is no battery and no system for harvesting energy. All electronic circuits require an energy source and a lack of an obvious system for powering the device is a problem. 2. There are no components such as integrated circuits, transistors, capacitors, diodes or inductive devices that would be required to create a "circuit" or "chip". A "chip" is not just a random configuration of tracks. The tracks are there to transfer electricity between components that shape and switch the electric current according to purpose but given that there are no components what are the tracks for? 3. There are 4 magnets. Probably neodymium. They produce a constant magnetic field. They do not generate "frequencies". The device sticks to a metal wall like a fridge magnet and doesn't vibrate. 5. The tracks are configured in such a way that even if components were attached at the "solder points" nothing would happen because the tracks are all shorted together. Electricity always takes the easiest path. If all the tracks are are shorted then the components will receive no energy input. 6. After testing with a multimeter I found that the tracks on the "circuit board" do not conduct electricity. If the tracks do not conduct electricity there is no possibility of transferring energy to components. ( there are no components ) 7. The magnets are isolated from the "tracks" and each other by a plastic layer and glue. It is not clear what the relationship the position of the magnets to the tracks might be. 8. There is no NVRAM, magnetic storage, optical storage, ROMs or other known systems for storing information. So claims from PowerInsole that they load information onto the device is difficult to comprehend. 9. There is no crystal or LC circuit to drive an oscillator. Even if there was there is no battery to drive it and the tracks are all shorted and the tracks do not conduct electricity. Given the above observations I find it difficult to believe that the device can function as advertised. What you essentially have is 4 small magnets on a printed card in a gel cushion for 69 euros. If I am wrong about any of the above I would be happy to have a respectful and open discussion about your technology. https://scontent-vie1-1.xx.fbcdn.net/v/t1.6435-9/163585013_10159594521424203_6934009271193208553_n.jpg?_nc_cat=107&ccb=1-3&_nc_sid=730e14&_nc_ohc=PA3Ximv76QEAX-4UVHb&tn=2GqubM3xHGE4RwnY&_nc_ht=scontent-vie1-1.xx&oh=4815887b32ff69023971b6c087ca6a5e&oe=60FA56D6 https://scontent-vie1-1.xx.fbcdn.net/v/t1.6435-9/163585013_1... https://scontent-vie1-1.xx.fbcdn.net/v/t1.6435-9/163740448_10159594521779203_136754634427714199_n.jpg?_nc_cat=107&ccb=1-3&_nc_sid=730e14&_nc_ohc=pF1ZeiPHaD0AX9Xu7ON&tn=2GqubM3xHGE4RwnY&_nc_ht=scontent-vie1-1.xx&oh=088db43f0fcb8d8d70494b487ed82f1d&oe=60FBBFAD https://scontent-vie1-1.xx.fbcdn.net/v/t1.6435-9/163740448_1... https://scontent-vie1-1.xx.fbcdn.net/v/t1.6435-9/164072709_10159594522179203_3139869476428661360_n.jpg?_nc_cat=100&ccb=1-3&_nc_sid=730e14&_nc_ohc=8qzT_wf-gGAAX_yfa7Y&_nc_ht=scontent-vie1-1.xx&oh=3f418250a489108b6d7dfe4e4ee60abd&oe=60FB64CA https://scontent-vie1-1.xx.fbcdn.net/v/t1.6435-9/164072709_1... https://scontent-vie1-1.xx.fbcdn.net/v/t1.6435-9/164005668_10159594574679203_6347388532427347427_n.jpg?_nc_cat=102&ccb=1-3&_nc_sid=730e14&_nc_ohc=jr4XO4IW-rEAX-NAEA8&_nc_ht=scontent-vie1-1.xx&oh=53909ec266de1dff59d3bba3e1e19919&oe=60FB063D https://scontent-vie1-1.xx.fbcdn.net/v/t1.6435-9/164005668_1... https://scontent-vie1-1.xx.fbcdn.net/v/t1.6435-9/164045289_10159594575179203_1420688379258392817_n.jpg?_nc_cat=108&ccb=1-3&_nc_sid=730e14&_nc_ohc=6bxIGxK31BUAX9g-Vbn&_nc_ht=scontent-vie1-1.xx&oh=97e5a1441f40482e77a3393e7111bd7e&oe=60FAACFA https://scontent-vie1-1.xx.fbcdn.net/v/t1.6435-9/164045289_1...
• ta988 5y ago
  When you want to do a proper work, your grants and papers get rejected because they are not innovatove enough or don't go far enough. So it is not a surprise that people that lied in their applications about what they can realistically do also lie when it comes to reporting results. Unfortunately there is no way out. I stopped counting how many reviewers of my grants disagreed on what was proposed, one saying that it was not innovative, the other saying that is was too risky to use this approach. We have a big problem in science, peer-review is broken and everything relies on it. And many reviewers are way out of touch about what happens in their field, I see reviews that clearly show the reviewer was sleeping for the last 10 years.
  • jerf 5y ago
    Worse yet, it compounds. The people approving grants, seeing all these amazing results promised, will then raise the bar for what kind of results you're promising. Which means the next batch of promises will need to be that much more extreme to get approved. It's a race to the top... or the bottom, depending on your point of view.
  • ethanbond 5y ago
    Yep. The incentives in science are all wrong. To maximize your chances of publication (i.e. keeping your job), you have to make the most outlandish claims you can possibly maybe defend. Additionally, the complexity of data/analysis is increasing every day while also the esoteric domain knowledge required to make any progress is deeper and more specialized.
    • BitwiseFool 5y ago
      Not enough people realize that science and academia are just as prone to organizational politics and corruption as everything else. Peer reviewed studies are great, but, just because it was published doesn't mean it represents "The Truth". And sadly, being skeptical of studies makes you appear less credible in arguments.
      • zepto 5y ago
• hirundo 5y ago
  I doubt most people understand the intensity of the incentive to mess with the data. In college I was a lab assistant for a professor who taught courses on research integrity and how to evaluate the quality of scientific papers. After thousands of hours of work on a study with routine 20 hour days collecting data, he wasn't getting what he needed to publish. At the tail end of one of those days I caught him with the equivalent of his thumb on the scale. He gave an excuse that he would have failed as an answer on one of his own tests. I argued a bit but then shut up. I kept shut up while that data was not excluded from the analysis that was eventually published. It wasn't enough to change the result, but still bothered me. So yeah, trust maybe but verify definitely. The rewards for faking it are just too great for an honor system to be reliable.
  • diob 5y ago
    I think it's more that the punishment for not faking it is too great. We need to be okay with following the scientific method and rewarding folks regardless of the outcome. Otherwise we're bound to see everything "succeed".
• vixen99 5y ago
  I wonder if this will apply to research published by pharmaceutical companies with a proven history of illegal activity in support of the products they release.
• dpatru 5y ago
  As shown both in the article and in the comments, scientific establishment seems content to tolerate fraud. But when research goes against big money interests, suddenly standards become very strict. See how Andrew Wakefield was treated. https://www.bmj.com/content/342/bmj.c7452 https://www.bmj.com/content/342/bmj.c7452
  • Animats 5y ago
    There's the famous story of the doctor who figured out, in the 1980s, how to cure stomach ulcers.[1] Stomach ulcers are usually a bacterial disease, and antibiotics work. The microbiologists in Brussels loved it, and by March of 1983 I was incredibly confident. During that year Robin and I wrote the full paper. But everything was rejected. Whenever we presented our stuff to gastroenterologists, we got the same campaign of negativism. I had this discovery that could undermine a $3 billion industry, not just the drugs but the entire field of endoscopy. Every gastroenterologist was doing 20 or 30 patients a week who might have ulcers, and 25 percent of them would. Because it was a recurring disease that you could never cure, the patients kept coming back. And here I was handing it on a platter to the infectious-disease guys. In 2005, he got a Nobel Prize. [1] https://www.discovermagazine.com/health/the-doctor-who-drank-infectious-broth-gave-himself-an-ulcer-and-solved-a-medical-mystery https://www.discovermagazine.com/health/the-doctor-who-drank...
    • rubatuga 5y ago
      Sounds like ivermectin is going through something similar. This off-patent, cheap, and safe drug is able to significantly reduce COVID-19 symptoms [0], and is extremely likely to be a potent prophylactic and treatment for COVID-19 [1], to the extent that the third wave in North America wouldn't have happened if it was used. The inventor, Satoshi Omura, won a Nobel Prize for inventing the drug in 2015, and tried to convince Merck (the original manufacturer) for many months to conduct an ivermectin trial for COVID-19, to no avail. On July 1st, he finally found a Japanese company called Kowa to charitably conduct a clinical trial for ivermectin, without Merck's help. Amazing right? But what response does Omura get from Western media? Quickly, his announcement video was deleted from YouTube [2]. You cannot find any English news about the Kowa trial being conducted. A few days ago, Omura was interviewed about ivermectin for the first time, on Yahoo Japan News [3]. Quoting him (using non-ideal Google Translate) > "My impression of WHO is that I feel sorry for being caught in a dilemma. Until now, I have only seen bright light in my life as a researcher. But this time, I learned for the first time after reading this article that shadows also exist in the world... Ivermectin is no longer a scientific issue, but a political issue." --Satoshi Omura It seems like big tech's misinformation crusade is biting us and science in the ass. [0]: https://pubmed.ncbi.nlm.nih.gov/33495752/ https://pubmed.ncbi.nlm.nih.gov/33495752/ [1]: https://academic.oup.com/ofid/advance-article/doi/10.1093/ofid/ofab358/6316214 https://academic.oup.com/ofid/advance-article/doi/10.1093/of... [2]: https://twitter.com/michaelcapuzzo/status/1410626769166733317 https://twitter.com/michaelcapuzzo/status/141062676916673331... [3]: https://news.yahoo.co.jp/articles/92c85ad9476f56a6fc51daec5654cb57b184fa75 https://news.yahoo.co.jp/articles/92c85ad9476f56a6fc51daec56...
• jgeada 5y ago
  It is about time to focus on the right problem: management standards that cause this crap to be pushed, and the effective immunity from consequences companies have when they lie. Why blame scientists when power is actually with management? And why let management and investors get away with this? It is about time "limited" liability had a pass through liability for this type of stuff: if you lie, knowingly or not, there are consequences and the consequences bypass limited liability. I bet if that happens this type of crap would immediately cease!
  • nathanaldensr 5y ago
    Unfortunately, you yourself are assuming that the actors in charge of holding people are accountable are themselves trustworthy. I feel we are approaching a singularity of low trust between people. It's only getting worse. You can't trust the watchers (journals) and you can trust the watchers' watchers (governments and the law). You can only trust yourself at the end of the day.
• ajsnigrutin 5y ago
  Another problem with studies is, that negative results are rarely published unless it's something really really "interesting". "we tried treating X with Y, and it didn't help (even though in theory it should have some effect)" is harder to get published than "we treated X with Z in vitro and it killed all the cancer cells (and noncancer ones too, whoops)".
• sharikone 5y ago
  I was terrified approximately 14 months ago when the big pharma companies started to get billions to develop the vaccine. My fear was that their research was fraudulent and they would be exposed and discredit science as a whole. I am very happy that this did not happen. I guess we are not in a total catastrophe situation but there is significant rot.
  • throwawayboise 5y ago
    That would never happen. Too much political capital is invested in the success of the vaccines.
• dleslie 5y ago
  Shouldn't all research be considered _tenuous_ until it is corroborated by a third-party without financial or social connections to the original researchers?
  • vadansky 5y ago
    Not even that is enough because you might end up with the filling drawer problem. For example you might end up with 50 people trying to replicate it, 49 failing and 1 "succeeding" by chance. The 49 won't publish because they probably got the "wrong" answer by chance, and they don't want to publish negative results, so they will leave it in the filing drawers. The one that did "replicate" it will publish and then in round 2 there is even more pressure not to publish negative results because "hey, this was replicated before already"
• xyzzy21 5y ago
  Sadly true. And especially government agencies and non-profits related to health.
• deleted 5y ago
  [deleted]
• giantg2 5y ago
  I think just as importantly is reining in the misapplication of studies. Too often I see some news/blog/politician/other say some policy or fact is proven by a study only to find that the study doesn't mean what they are claiming. This can be stuff like using animal studies not supported by further human studies and claiming the effect is true for humans. Or confusing correlation for causation. Or viewing the speculated application of the study found in the conclusion to be absolute truth when many times the authors themselves claim additional studies would be needed to evaluate other aspects or confirm their findings. A classic example was gender wage gap misrepresentation about 6-8 years ago. Many news groups and even the president were misinterpreting (and spreading that misinformation about) the BLS study to mean that a man and a woman in the same job with all else equal, the woman would only make $.80 on the dollar, when in fact the issue is an aggregate level issue mostly due to structural issues (and require different remedies than proposed). At least it seems many places have since realized their mistake, yet the misinformation persists in the general public.
• enriquto 5y ago
  why just health research? all science is already based on that very principle. In fact, a stronger one: you cannot ever prove that some research is "true", only that no inconsistency has yet been observed.
  • throwawayboise 5y ago
    Replicating the results would be evidence that the original research was "true." Unfortunately nobody wants to fund research that only attempts to replicate an already published result, when they could spend the same money on novel research.
• marsven_422 5y ago
  This just in : Scientists are just as human as everyone else.
• miga 5y ago
  No, just a statistical reality of multiple hypothesis testing. Just like you wait a few blocks for a confirmation on a blockchain, you have more and more confidence with confirmation of health research by independent papers. See statistical analysis in Ioannidis' "Why most research findings are false": https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo...
• 0xcde4c3db 5y ago
  I knew there were issues with various kinds of research. Things like p-hacking, "touching up" data, and so on. But the lead example is pretty wild: > As he described in a webinar last week, Ian Roberts, professor of epidemiology at the London School of Hygiene & Tropical Medicine, began to have doubts about the honest reporting of trials after a colleague asked if he knew that his systematic review showing the mannitol halved death from head injury was based on trials that had never happened. He didn’t, but he set about investigating the trials and confirmed that they hadn’t ever happened. They all had a lead author who purported to come from an institution that didn’t exist and who killed himself a few years later. The trials were all published in prestigious neurosurgery journals and had multiple co-authors. None of the co-authors had contributed patients to the trials, and some didn’t know that they were co-authors until after the trials were published.
  • wolverine876 5y ago
    It's one example, chosen and presented by someone with something to prove, and which fails to provide any evidence (such as the names of the studies or lead author).
    • sjwalter 5y ago
      There are plenty of other good examples. Try this one: https://www.the-scientist.com/features/the-surgisphere-scandal-what-went-wrong--67955 https://www.the-scientist.com/features/the-surgisphere-scand... "It sounds absurd that an obscure US company with a hastily constructed website could have driven international health policy and brought major clinical trials to a halt within the span of a few weeks. Yet that’s what happened earlier this year, when Illinois-based Surgisphere Corporation began a publishing spree that would trigger one of the largest scientific scandals of the COVID-19 pandemic to date. "At the heart of the deception was a paper published in The Lancet on May 22 that suggested hydroxychloroquine, an antimalarial drug promoted by US President Donald Trump and others as a therapy for COVID-19, was associated with an increased risk of death in patients hospitalized with the disease." They completely fabricated the data. To hurt Trump.
• supperburg 5y ago
  People cite paper titles like they are facts. Nobody even knows whether or not it was an epidemiology study or an interventional study, they just say “they did a study showing that underwater basket weaving lowers your risk for colon cancer!” Nobody actually reads the studies. During a debate or conversation, many people cite studies in support of their point of view. This is a problem because now the other person is swamped with a dozen studies to analyze and debunk before proving he’s right. And academia is producing huge volumes of these bullshit studies so no matter how you slice it, a huge unnecessary burden has been created of digging through all of them and circling the flaws. Thankfully, there is an emerging cultural mechanism to deal with this in the growing “epidemiology is bullshit” sentiment. This is good because it reduces the bulk of bullshit that will ultimately need to be processed and debunked. If the study is epidemiology just cross it out by default. Those studies need to burn in hell. Shine the light of day on them and brandish the holy crucifix whenever you see one. The only thing worse than a science denier is a person who blindly parrots study titles without ever reading the body of the paper let alone understand it. People complain endlessly about armchair scientists who are spreading misinformation based on their uneducated assessment of scientific data. And the people who complain about this are always the same people who cite studies that they don’t understand like complete idiots, spreading misinformation just as widely.
  • wolverine876 5y ago
    > there is an emerging cultural mechanism to deal with this in the growing “epidemiology is bullshit” sentiment There is an emerging cultural mechanism - more a rampaging mob - that says 'X is bullshit' as a simple way of denying facts or issues that are inconvenient or difficult. It's used for the news media, academia, non-partisan government agencies (e.g., the CDC), etc., etc. and for everyone who disagrees. I say this social mechanism is bullshit - the sources they disregard come with plenty of evidence, saying they are bullshit comes with none - it's just easy to say. It's also very destructive. Where do we get our epidemiology or news or whatever else if anyone can claim anything is bullshit at any time, halting everything until they are proven wrong? It's up to them to prove their claim right; we can't all halt and freeze in place every time someone makes the minimal effort to vocalize, 'X is bullshit'. Without evidence, their claim is meaningless and should be ignored. Epidemiology, the imperfect human institution it is, provides many successes.
• wxnx 5y ago
  I work on large multi-center clinical trials as a machine learning engineer. One of my projects involves the semi-automation of the detection of fraudulent data. There's one link in the chain here missing that some people here seem to be ignoring. The authors of this post (while entirely correct) draw no link between "bad data" (which is doubtlessly responsible for a large number of "bad papers"/"bad trials") and "bad clinical practice." I don't know a single clinician who would base their care on the findings of a single-center RCT of the kind described in this article. Or the findings of a meta-analysis of single-center RCTs, for that matter. Bad data happens in multi-center RCTs too, and in fact that's what I'm focused on, but a lot of work already (and therefore $, for the cynical) goes into the validation of data (see [1] for a brief description). Phase III clinical trials in the west practically require a robust multi-center RCT, where systemic fraud is very difficult to perform (but not impossible [2]). By the time a Phase III trial is conducted, the efficacy of the drug can already be estimated, and the focus of the drug company (who yes, often fund these trials) is to conduct a trial which is unimpeachable in the face of a regulatory board (who are generally good at their jobs, although the revolving-door tends to reduce public trust and should be legislated away). In short, I support most of the proposed changes to incentives around publish-or-perish. I reject the notion that these incentives are (currently) significant drivers of decreased quality of standard of care in the West. I think global governance structures, as suggested in this article, could improve understanding among both clinicians who are not necessarily scientists and the general public about just how validated a given standard of care is. tl;dr Most good evidence-based practitioners already think this way -- not because they inherently believe fraud is rampant, necessarily, but because evidence says the kinds of studies where fraud is most prevalent are untrustworthy for other reasons. [1] doi:10.1177/1740774512447898 [2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4340084/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4340084/
  • SubiculumCode 5y ago
    Some segments on HN have strong anti-scientist sentiments (even while they proclaim to be pro-science), assumine we all are crooked, stupid or both, hence your insightful and reasonable comment being downvoted. Is there fraud? Sure. Is there a lot of fraud happening in American science? I don't think so. To quote the article: "Many of the trials came from the same countries (Egypt, China, India, Iran, Japan, South Korea, and Turkey), and when John Ioannidis, a professor at Stanford University, examined individual patient data from trials submitted from those countries to Anaesthesia during a year he found that many were false: 100% (7/7) in Egypt; 75% (3/ 4) in Iran; 54% (7/13) in India; 46% (22/48) in China; 40% (2/5) in Turkey; 25% (5/20) in South Korea; and 18% (2/11) in Japan. Most of the trials were zombies. Ioannidis concluded that there are hundreds of thousands of zombie trials published from those countries alone. "
• Ajay-p 5y ago
  Is the incentive for publishing research that is fraudulent primarily money, prestige, or are there just that many professors that graduate students required to publish something?
• NoblePublius 5y ago
  “He is now sceptical about all systematic reviews, particularly those that are mostly reviews of multiple small trials.” This describes basically all “meta analyses” of Covid claims, especially the effectiveness of masking and lockdown. This is at the heart of “epidemiology” which is, as far as I can tell, the science of organizing data to meet predetermined political demands.
• fasteddie31003 5y ago
  I've had a thought on how to solve this issue by using basically a research paper futures market. You could implement this with Ethereum Smart Contracts. You have a market around the validity of research papers. You would need some authority that would act as the oracle of the paper's validity. If the paper is found invalid before some time period the people who bet on the paper's truth would lose their money to the people skeptical of a paper. This would also act as a mechanism to signal which papers people don't trust by the amount of people betting against a paper's validity.
  • mrits 5y ago
    This could actually make it worse. Companies could pump their own research anonymously
    • Noumenon72 5y ago
      That could still be a good system, like pay-for-play on the radio. While you could pump complete crap, it's more profitable to pump songs that people will actually like, and companies with better songs end up with more money to pump stuff.
• ChrisMarshallNY 5y ago
  I'm actually kind of flabbergasted that people -no matter who they are- are automatically given the benefit of the doubt, without question. I'll bet that a lot of folks just assume that anything they do will be taken at face value, without question or inspection. I also suspect that many "brought and paid for studies" are done this way. For my own work, I generally assume that most of these studies are pretty much worthless, and tend to do some of my own homework before accepting them. Since most don't concern me at all; it's not a big deal. Health is just one place this kind of thing happens. Software Development is absolutely rife with bad implementations. I am not in AI, but I have heard from a number of people that AI has a big problem with irreproducible results. https://web.archive.org/web/20190926055757/http://www.jir.com/ https://web.archive.org/web/20190926055757/http://www.jir.co...
  • version_five 5y ago
    I work in ML research and I used to do experimental physics. I'd agree that specific results in papers can be hard or impossible to reproduce, but that never really bothers me because at least in my work, the specific experimental result is rarely material to why I'm interested in the paper. It's more of a demo, and like a demo, you know its orchestrated to look good. What I'm interested in is what is the mechanism behind the advance and do I think its applicable or relevant to what I'm doing. If the paper is really just a random observation of something that worked better, without a causal explanation, it's not very interesting, but I don't see those often. Maybe health research is very different, and people are latchjng on to surprising results they find in papers, but I doubt it's a big problem in academia, much more likely in the media. If I was a doctor and saw an out of the blue study claiming a surprising result, I'd discount it accordingly. If I saw a causal explanation with evidence, I'd give it closer scrutiny and follow up if it seemed relevant to me. That is how research works in my experience.
• ineedasername 5y ago
  he set about investigating the trials and confirmed that they hadn’t ever happened. They all had a lead author who purported to come from an institution that didn’t exist To me, this doesn't mean that simple distrust is the answer. These are basic issues that should be revealed with even minimal due diligence during the editorial & peer review process. Peer reviewers and Journal Editors should brink a skeptical mindset to article submissions from the outset before they're ever accepted for publication. After that? Well, whenever research is on an emerging topic there is a certain amount of scientific skepticism you should use. Same if results go against an established consensus on a topic. However this is where the "replication problem" enters the picture because replicating research has a lower status. When it comes to media reporting, things get even more complicated. New science is messy. You only have to look at COVID research for the past 1.5 years, and when it's an issue of such public urgency, EVERY development hits the public eye, pulling back the curtain on the sausage factor. Because new science is rarely "Hey look what I discovered!" followed by "Yay we all agree!" It's more of a conversation or dialectic, with ever more research revealing the picture a bit more until there's enough to be confidence in a given interpretation. And even there, work proceeds on alternatives. The above is very much NOT how science is taught to the public in schools. You learn "Darwin Discovered Evolution!", not the significant years-long process of researchers arguing it out, sometimes even with heated vitriol. You learn "Newton Discovered Gravity!", not all of the complexities and disagreements that continue even to today. Out education systems have failed society when it comes to truly understanding the scientific process. This is why distrust of science increased. Because in past decades awareness of scientific advances often only reached the public after at least part of the sausage was made, meaning now it looks like it's descended into total disarray.
• mcguire 5y ago
  "...Ian Roberts, professor of epidemiology at the London School of Hygiene & Tropical Medicine, began to have doubts about the honest reporting of trials after a colleague asked if he knew that his systematic review showing the mannitol halved death from head injury was based on trials that had never happened. He didn’t, but he set about investigating the trials and confirmed that they hadn’t ever happened. They all had a lead author who purported to come from an institution that didn’t exist and who killed himself a few years later. The trials were all published in prestigious neurosurgery journals and had multiple co-authors. None of the co-authors had contributed patients to the trials, and some didn’t know that they were co-authors until after the trials were published. When Roberts contacted one of the journals the editor responded that “I wouldn’t trust the data.” Why, Roberts wondered, did he publish the trial? None of the trials have been retracted." I realize that meta-analysis is regarded as a valid research method, if not one of the best, but honestly, I don't know why. If the original studies are garbage, no amount of statistical manipulation is going to make them not-garbage.
  • FeteCommuniste 5y ago
    Meta-analysis normally tries to exclude "low-quality" studies but if the standard of honesty in a field or sub-field is truly abysmal, I guess it's GIGO.
  • metalliqaz 5y ago
    well if the original studies are inadequate simply because N is too low, then grouping many of them can resolve that problem.
    • mahogany 5y ago
      Is that really true from a statistical standpoint? It seems to me that running ten 20-person studies is different and less valuable than running a single 200 person study, because each of the 20-person studies has a much larger error range that you have to account for. But I'm also not good with stats.
      • 5y ago
• RandomLensman 5y ago
  I think the first important question to ask is: is the research I am looking at directly applicable and relevant to other people? If stuff is highly applicable and relevant, then the chance to get away with made up rubbish starts to drop quite a bit, because people will want to try for themselves - and fail. It gets more problematic when the application takes a long time to show effects, i.e. long observation periods - there more can be faked. Most published research is largely inconsequential and not interesting to but a tiny few, so "fraud" can easily hide in there. Yes, it increases the sum total of knowledge, but that might be about it. I am not saying most research isn't done well, just that there isn't a way to assess correctness at scale and only high-profile results might get a fast check (and even there...)
• swayvil 5y ago
  If lying is the more profitable course then you can trust that it's a lie. Is that capitalism? Is that what we're looking at here?
  • UnFleshedOne 5y ago
    It's humans. You should look at how research was done in communist countries or anywhere with strong ideology. People are built to game systems.
    • swayvil 5y ago
      Well we're also built to behave in accordance with what's "right" (moral etc). So options are being weighed. The "immoralizer" must not be "promise of profit" (unless you are a total lizard) but rather freedom from anxiety, promise of safety.
• aaron695 5y ago
  Feels like the gun lobby pushing blame to computer games. Everyone knows explicit fraud has nothing to do with it. It exists because academia is broken and can't self correct.
• FL33TW00D 5y ago
  This is why I left academia.
• AlexCoventry 5y ago
  The video is worth watching. It goes into much more detail about just how extensive, brazen and destructive the fraud has been. https://www.youtube.com/watch?v=3BXEOey62O4 https://www.youtube.com/watch?v=3BXEOey62O4
• kobieyc 5y ago
  Fraud vs incompetence is an important dimension here. I think a lot of the time people are just incompetent. In fact I assume that > 80% of scientific papers in fields with high levels of environmental noise (social science, environment science, medical, etc) are bad science. So TL;DR I just assume incompetence instead of fraud.
• briantakita 5y ago
  A major investor & leader in healthcare that shall remain unnamed had a book "How to Lie with Statistics" in a public reading list. It's a good book & a quick read. Highly recommended. It's interesting that it takes an editorial to make people suspicious of statistics, how statistics can be abused, & the conflicts of interests that many people who utilize statistics have. Sample bias needs to be treated as deliberate dishonesty rather than a simple mistake. These people who make these mistakes are professionals and should know better. Their code of conduct should penalize them harshly for making these sort of mistakes. A strict code of conduct with harsh professional penalties are necessary to remove bad actors who hide behind subtle lies that have a major impact on public policy & public opinion. A slap on the wrist means it's always worthwhile to lie with statistics. A removal of license & banishment from the profession on the 1st or 2nd offense would quickly remove the bad actors. This code of conduct should also extend to the peer review process. If the peers pass bad statistics, the peers need to be held accountable as well.
• sjwalter 5y ago
  "The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness." Richard Horton, current editor of The Lancet: https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(15)60696-1/fulltext https://www.thelancet.com/journals/lancet/article/PIIS0140-6...
  • sjwalter 5y ago
    This is also especially ironic considering The Lancet published a totally fabricated study that supposedly demonstrated how dangerous a Trump-touted Covid treatment was. https://www.the-scientist.com/features/the-surgisphere-scandal-what-went-wrong--67955 https://www.the-scientist.com/features/the-surgisphere-scand...
    • nimish 5y ago
      The Lancet is no stranger to promoting pseudoscience. John Snow challenging the miasma theory of cholera really ground its gears. Plus ça change...
      • magnetowasright 5y ago
        Can’t forget the Andrew Wakefield vaccine nonsense they platformed as well
  • brutusborn 5y ago
    He talks about the case against science as well as bad studies like they are the same thing. Science is simply a method that can be done well or badly. The bad studies aren't fundamental to science itself.
    • zepto 5y ago
      > Science is simply a method that can be done well or badly Science as a word doesn’t typically denote “the scientific method”. It usually means the body of knowledge and the collective of institutions that claim to practice the method. The bad studies may not be fundamental to some abstract ideal of what science could be, but they seem pretty fundamental to humans doing science in the real world.
• wolverine876 5y ago
  The headline is more than a bit sensationalist. I never know what to make of BMJ, which sometimes seems sensationalist: Can anyone in the industry or profession characterize who they are, what they do, and what their reputation is?
• kazinator 5y ago
  Betteridge's Law of headlines says: no! We can be skeptical of researchers' results without the hostility of assuming that they are guilty of fraud until proven innocent.
• i_left_work 5y ago
  From my experience, most of it is. I just left a high paying position working in the healthcare space as a data scientist, because it became clear this was known and there was no intention to improve the situation. Instead, the focus was on selling and making a quick exit.
• tarere 5y ago
  I will address the elephant in the room on this one. 168 comments so far, I did a research for word "vaccine" in the page, not a single result. So I assume this has not been discussed despite the current situation we are living right now. I've just read 2 articles about Moderna in general and Bancel in particular : https://www.bloomberg.com/news/features/2021-07-14/moderna-mrna-targets-hiv-cancer-flu-zika-after-covid-vaccine https://www.bloomberg.com/news/features/2021-07-14/moderna-m... And Vanity Fair, unfortunatly in french https://www.vanityfair.fr/pouvoir/business/story/stephane-bancel-et-moderna-therapeutics-lentreprise-qui-a-promis-a-donald-trump-un-vaccin-contre-le-coronavirus/12140 https://www.vanityfair.fr/pouvoir/business/story/stephane-ba... This is a frightenning read as we people in France are being literraly forced by our gouvernement to be vaccinated (Moderna, Pfizer) or be socially terminated. I did not know, I'm pretty sure 99,99% of people also don't, that Moderna did not release a single product before 2020 and it "all in" bet in the Covid Vaccine with up to 1 billion funding from operation warp speed. Moderna was sometimes refered as the next Theranos. His leader fits clearly in the sociopath territory of Silicon Valley tranhsumanist billionnaires. "risk very big, win very big" is his mantra, this man wants to vaccinate billions of people annualy (read article please). More, if you read those two articles, you could change Vaccine by any software product and you would have a typical business article about a Silicon Valley startup. This is frightening to death to think that this technology "software vaccine" is to be used on the whole population with only a few month of study and, worse, with "forced consentment" on populations. Guess what happens when money conflict with health.
  • nabla9 5y ago
    No results because vaccines are not accepted based on publications and peer review on scientific papers. They go trough completely different and extremely rigorous testing process where everything is documented carefully, documents are examined and double checked. It's great to see 168 comments before first anti-vaxxer comment.
    • tarere 5y ago
      I was speaking of the HN page discussion, but it seems my search on Firefox is completely broken, so I withdraw my mention of no vaccine reference, but I stand to be commented on the subject.
• ghoward 5y ago
  My thoughts: https://gavinhoward.com/2019/12/replication-and-retraction-crises-in-science/ https://gavinhoward.com/2019/12/replication-and-retraction-c... .
• sjwalter 5y ago
  What I want to know is how does this issue impact the notion that we all seem to buy into that we should "follow the science". Scientists themselves have a hard time "following the science". Add to it the observation that when an issue is getting lots of attention outside of academia, then there are usually some really strong incentives (profit, prestige) associated with doing the science and applying it (e.g., epidemiological science during a global pandemic). The question seems not to be about how can normal people "follow the science" but rather, why should normal people trust at all that any touted science is anything more than bullshit spouted by highly-motivated sophists?
  • kingkawn 5y ago
    "follow science" is just another way of saying "do as we say not as we do."
  • nitwit005 5y ago
    I would assume the people saying "follow the science" generally don't mean "believe recent research publications". I still occasionally see things like "hanging a potato to your wall will cure your child's flu" being debated by friends of friends on Facebook. You'd need to take a time machine several hundred years back for it to be within the realm of realm of genuine scientific debate.
    • hallway_monitor 5y ago
      You seem to be indicating the preferred time window for which research to trust. Not too new, not too old. Not the worst algorithm you could choose, and I agree. This is why I stay away from drugs, procedures, and any kind of guidance from the medical profession that is less than 20 years old.
      • chitowneats 5y ago
        This is exactly how I feel after being prescribed a drug as an adolescent that in fact made my condition worse. It was a very commonly prescribed anti-depressant that was eventually found to increase the risk of suicide in those under 18 years old. I was on the drug during the first decade after its release. It is no longer prescribed to minors.
• alanbernstein 5y ago
  The "time to assume" this was at least 15 years ago: https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...
• cheaprentalyeti 5y ago
  Well, while you're at it, can you tell me which side is telling the truth about Remdesivir, the US health care bureaucracy or the WHO?
• trompetenaccoun 5y ago
  Only 20%? That seems surprisingly low, I'd have thought it's even worse than that.
• wly_cdgr 5y ago
  Why only health research? The prudent and sensible person never puts more trust than circumstances require in anything they haven't independently verified
• pedalpete 5y ago
  I thought the article was going to be about research that could not be reproduced, not outright fraud! We're in the sleeptech space, so now I understand why there is so much snake oil that has university names attached to the "research". I thought it was just the company that was selling the product that was a fraud, not the research institutes themselves!
• whiddershins 5y ago
  Always.
• motohagiography 5y ago
  > Researchers progress by publishing research, and because the publication system is built on trust and peer review is not designed to detect fraud it is easy to publish fraudulent research. The business model of journals and publishers depends on publishing, preferably lots of studies as cheaply as possible. They have little incentive to check for fraud and a positive disincentive to experience reputational damage—and possibly legal risk—from retracting studies. Funders, universities, and other research institutions similarly have incentives to fund and publish studies and disincentives to make a fuss about fraudulent research they may have funded or had undertaken in their institution—perhaps by one of their star researchers. Regulators often lack the legal standing and the resources to respond to what is clearly extensive fraud, recognising that proving a study to be fraudulent (as opposed to suspecting it of being fraudulent) is a skilled, complex, and time consuming process. Another problem is that research is increasingly international with participants from many institutions in many countries: who then takes on the unenviable task of investigating fraud? Science really needs global governance. Long excerpt but the best arguments defy paraphrasing. Arguably, Science would benefit more from decentralization than global governance, because more science could get done instead of spending time on the zero sum politicking of working the governance systems. When you train people to in-effect litigate their research as a case for approval of committees, the falsification of everything is an unavoidable outcome. It becomes like a legal dispute, where there is no truth, just the prosecution of their case to make anything stick they think they can. It's not just health research, it's a couple generations of graduates who were given a simple blunt instrument in a system incentivised to popularize the use of that dull tool. The tool reduces to, "there is no truth, only power, words are just tools to struggle for it, responsibility is what you have when you don't have power, and if you take care of this system, it will take care of you, but if you call it out it will come for you first." That's the One Big Thing the hedgehog knows. If you practice it, you can rise to the top of almost anything without knowing much about it. The way to beat it is to set a bar of competence and concreteness. Scrutiny of experimental data goes a long way to setting that bar.
• hatware 5y ago
  Welcome to the club
• rob_c 5y ago
  I had the "privilege" of reviewing a certain piece of software written to model a certain pandemic outbreak... Numerical modelling in biology/virology feels rife with serious problems that are simply sidestepped because of ego and status. I raised alarms and as merely a senior expert in distributed computation and numerical modelling I was shutdown quite strongly. After the group in question were forced to accept our published work refuting their results to be FUBAR due to severe numerical instabilities which hadn't even been checked for we were shut out for a group who patted them on the back for being "oh so clever"... (our published work numerically demonstrated results they were presenting as fact were compatible with statistical noise and that it is known they didn't have the required extra 100xcompute or 100xtime to produce results as precise as what they were after) Had a similar experience a few years ago with a geological modelling group and a group hoping to do biochemistry on a huge cluster of 20k cpu cores. For all of the "numerical sciences" achievements only really physics and chemistry (and maths) seem to have championed the ideas of reproducibility and falsifiability in numerical analysis and simulation.
• magnetowasright 5y ago
  As a layperson that feels effected by this issue (as I’m Autistic) I feel powerless, both to really understand this at depth, or how to do anything to fix this. Advocacy is moving things forward but I don’t know if that’s managed to reach researchers or effect the kind and quality of research being done. Luckily some of the more overtly harmful trash ‘research’ that underpins a lot of the medical model and (lack of) understanding about Autism is being dismissed/is now somewhat regarded as contentious (Baron-Cohen’s awful ‘Theory of Mind’), but the harm is done, and things are very slow to change. The dialogue and research around queer issues went through a lot of changes like homosexuality eventually being removed from the DSM and the trans related stuff has been getting better. How do we make progress like that for Autism? More Autistic researchers researching Autism? Actually involving Autistic people is a must. My comment is a little messy, and I don’t really know what I’m talking about, I’m just wanting to voice a concern from a different perspective.
• rituraj_tiwari 5y ago
  All research grants should take 40% of funds and give them to a completely separate team to do "QA" research. This will give up and coming researchers cred to get primary funding after a few good QA projects. If research cannot be reproduced by an independent QA team, it does not make sense to fund it or trust it.
• 13daug 5y ago
  Yes everybody should stay healthy best Treatments
• satellite2 5y ago
  I'm going to go against the general concensus that seems to be forming here. In my experience, at any scale, you cannot resolve trust issues by reducing trust. When you reduce trust, people producing the science expect to get more scrutinized and determine they can afford to get less critical of their own work. The first thing I saw in academia is that scientists are almost never dishonest. Science is a complicated psychological process as you need hope to guide your research and you need rigor to test your hypothesis. When you let hope take a too big spot in your mind you tend to publish your partially tested hypothesis as results and when you lack it you tend to get paralized or to work on predictable work. And I saw this process multiple times. But I extremely rarely saw actual malice. Even when there was an incentive to produce bad science, you could see that the author genuinely convinced themselves that what they produced was riogorous and correct. So like in the political arena, it's a vicious circle to reduce trust. And in the contrary it's a virtuous one to increase it. When most of the science is considered genuine, it becomes a much harder hit on any scientist to get its work shown incorrect. That in turn promotes more self criticism which is the cheapeast and most effective kind of review.
  • throwaway81523 5y ago
    > The first thing I saw in academia is that scientists are almost never dishonest. Was that health research, or some other field? It's plausible that health research is more corrupt than, say, theoretical physics, because of the immense amounts of money that can be gained from particular outcomes.
• nazrulmum10 5y ago
  Nice article
• Schnurpel 5y ago
  In many European jurisdictions, including Austria, the losing party pays all costs of the lawsuit, including lawyers’ fees. This simple rule takes care of most frivolous suits. So the answer is: Go sue me.
• rbanffy 5y ago
  I believe we are getting to the point where we need to create a broad category of crimes for misleading the public. Form lying as a trusted public figure (elected politician, government employee) all the way to publishing or submitting fraudulent research for publication. Countless lives were lost in the past year because people believed fraudulent studies about COVID treatments. Worse than that, government officials publicly supported those false reports and increased the damage they caused for political or even financial motivations. The search for objective truth is a societal defence mechanism. Assaulting it is an assault on society itself.
  • l33tbro 5y ago
    Keep dreaming. Who's going to advocate for that on your behalf? A politician? An private or public institution complicit n this behaviour?
    • aww_dang 5y ago
      Agreed. Instead of looking for institutional solutions, we can demonstrate the need for critical thinking and skepticism as individuals. This is a can-do solution we can start with now. The above poster's plea of, "there ought to be a law" removes the agency of the individual. There are also serious problems with appointing fact checkers as impartial arbiters of objective truths. It is an untenable scheme. A naked appeal to authoritarianism.
      • rbanffy 5y ago
        > we can demonstrate the need for critical thinking Good luck with that. Humans are not always rational and don't always act on their best interests, much less on the whole species' best interests. If we leave saving our species to individuals, I wish the cockroaches better luck. I, personally, would bet on ants and bees, as they seem to be much better organized than us.
        • aww_dang 5y ago
          My alarm bells would go off if someone claimed to know what is best for the whole species. If individuals are not rational as you say, then how would politicians, technocrats or other central planners be rational? If an individual doesn't have the right to coerce you, how does a collective of individuals have the right to coerce you? Individualism is the decentralization of information and decision making. It has a different failure mode. If we accept that men are fallible, then individualism allows for a competition of solutions and ideas. Humans will never be perfect. Decentralization allows us to progress and iterate faster than central planning, which has all of the same problems with what you call "irrationality".
• morgengold 5y ago
  Incentivise amount of data for open questions. Not "inovativeness" or size of effect. A system with the wrong incentives will produce bad outcomes.
• Kosirich 5y ago
  Is there an equivalent of a 'whistleblower' prize in science? Where someone get's money and prestige from finding falsely reported studies, data ect.
• garfieldnate 5y ago
  Is it naive to suggest that we try something similar to 3rd party security audits? If a journal or institution submits itself to multiple 3rd party audits, which are allowed to test a variety of nasty tactics to get bunk science approved or published by them, it could potentially be a good way to keep them honest. Unless, of course, they collude and become rubber stamps (like certain food labels in the US).
• grumblenum 5y ago
  The article is about intentional error; however, I have found this to be very approachable and entertaining primer on (probably) unintentional error in research: https://www.statisticsdonewrong.com/ https://www.statisticsdonewrong.com/ Research that has been further filtered onto your favorite infotainment coffee-table-science column further suffers from the 'eigonvalue' problem. I think assuming intentional fraud is excessive for a default position, but complete trust of any document formatted with Latex because "it's science" is probably worse.
• boxslof 5y ago
  All this sounds rather unsettelling. I am left with many questions. I'll reduce them to two: 1. are there people here who (especially medical field) do have the feeling the current system of research does its job (on average)? 2. Applied to very hot topic of vaccine safety assessment for the new covid vaccines, how sure can we be there is/will be enough room to critically assess this? (sorry i had ask, but just read https://www.wired.com/story/the-cdc-owes-parents-better-messaging-on-the-vaccine-for-kids/ https://www.wired.com/story/the-cdc-owes-parents-better-mess... so yea...)
• NautilusWave 5y ago
  A bit off topic, but is anyone else tired seeing acrostic poems being passed off as acronyms?
• neycoda 5y ago
  20% wrong doesn't mean distrust by default... the question is answered in the first few sentences.
• CRConrad 5y ago
  From the end of TFA: > Research authorities insisted that fraud was rare, didn’t matter because science was self-correcting, and that no patients had suffered because of scientific fraud. Thus aiding and abetting murderers like Paolo Macchiarini. > All those reasons for not taking research fraud seriously have proved to be false, and, 40 years on from Lock’s concerns, we are realising that the problem is huge, the system encourages fraud, and we have no adequate way to respond. As his patients could attest... If they weren't dead. https://en.wikipedia.org/wiki/Paolo_Macchiarini https://en.wikipedia.org/wiki/Paolo_Macchiarini
• nolson 5y ago
  I presently work at Defense Innovation Unit where pretty much all we do is test and evaluate prototypes on behalf of the services and agencies. We spend hundreds of thousands to millions testing each thing. Because hundreds of millions of dollars may be spent based on what we find. We probably get 80 pitch decks for every area of interest we announce. That usually gets winnowed down to 5-10 that actually have technical merit and are responsive. We get a few shysters with every batch, but the vast majority are simply too optimistic about what they've done or what they can accomplish.