63 ms·

Medicine is plagued by untrustworthy clinical trials

333 points by headalgorithm 3y ago

gordian-not 3y ago
Masters and Phd requirements should include repeating another research and checking whether they can recreate the results
- sealeck 3y ago
  Unfortunately some of the experiments require a whole team to carry out and take a huge amount of time to set up as well as having to be carried out under very specific conditions.
- Slaminerag 3y ago
  My institution's been included in studies primarily for patient access. If we're not included, then good luck getting enough patients, and even then it can take several years to enroll enough patients. Replicating such a trial would be near impossible.
__666__ 3y ago
[flagged]
- wahnfrieden 3y ago
  Is it structural greed?
CrampusDestrus 3y ago
it's 2023, we have the means to cheaply record and store audio and video evidence for basically any medical experiment. we can record every patient reaction and opinion without relying on the reasearchers' hearsay. we also have the means to store and distribute all the binary/textual raw data gathered throughout the experiments. maybe as an intermediate step we could make available all the recordings to the peer reviewers and only offer the raw experimental data bundled in the paper publicly? maybe in the future we can have 1TB studies without breaking a sweat? maybe all the money we give to publishers can be spent on servers to archive all the primary data so at least we aren't simply filling the pockets of MBAs?
- jamesdwilson 3y ago
  how do you prevent cherry picking?
  - CrampusDestrus 3y ago
    that's such a vague question for example, if you have 50 partecipants but only provide the multimedia evidence for 20 of them your study should be thrown out the window
    - dolni 3y ago
      What is preventing someone from having 200 participants, but saying they only had 100 participants, and then only providing evidence for 100 participants?
      - CrampusDestrus 3y ago
        The researchers will have to have gone through some kind of third party agency to get the partecipants. This agency should be queried to see the number they report
        dolni 3y ago
        How does this agency determine who can meaningfully participate in the study? Are they going to have the expertise to make that determination for _every_ study that could possibly be conducted? What is the difference (to a layperson) between cherry-picking participants and rejecting participants because they do not meet your study's criteria? Who funds this agency? Do the members of the reviewing agency have their own biases, and might those biases tarnish the reputation of a study that is actually well-conducted? (hint: this already happens in journals)
      - throw9away6 3y ago
        You have to declare the study population before you request the next round of funding. Thereby fixing the problem.
Eumenes 3y ago
The over prescription of statins is a great example of what pharma sponsored clinical trials result in
- remote_phone 3y ago
  This. Statins claimed to be free from adverse reactions, but it turns out that about 30% of the participants were taken out of their clinical study because of “non-compliance”. However, if you dig in further, the non-compliance was because of adverse reactions. You can’t trust pharma companies if their data is secret.
  - TheBigSalad 3y ago
    There must be more to it. Everyone who drops out is scrutinized.
  - anonuser123456 3y ago
    If you dig further into those adverse reactions, you’ll find they are approximately equal to adverse reactions of placebo.
    - Madmallard 3y ago
      Except for the part where there is a clear mechanism and cause related to muscle damage and cellular dysfunction.
      - anonuser123456 3y ago
        Show me the double blinded, placebo controlled study.
        Madmallard 3y ago
        https://www.acc.org/about-acc/press-releases/2016/04/03/15/02/trial-offers-objective-evidence-of-muscle-related-side-effects-with-statins https://www.acc.org/about-acc/press-releases/2016/04/03/15/0... here you go anonuser123456 You will likely be able to find scientific studies toward any conclusion you want to make but there is objective evidence of this and in my own personal life significant anecdotal evidence. Given the OP title and topic and the massive reproducibility problem in medicine and perverse incentive problem in medicine, you will not be able to know the truth about the safety of medications to a high degree. It is like the reviews problem on Amazon. All medications come with studies giving 5 star reviews. But the reality is nowhere near the same. Blackbox warnings and such are all we have to place a significant ding on this problem before taking the drugs off the market for clearly causing intolerable issues in large swaths of people. This is a complex muddled issue. Argumentation with citations of 'articles' from journals likely will not lead to truthful discovery, unfortunately.
obblekk 3y ago
There should be Nutrition Facts but for scientific trials. Independent agency just publishing quality assessments of the trial. This should be an async non blocking evaluation. The statisticians who do it should be anonymous by default. There should be an appeals process for a scientist to explain why an unconventional new method is actually robust. There should not be a single number published by this process, but rather a list of stats that speak to the overall quality of the trial on many dimensions (power, sources of bias, etc). Only information that would not be the same on 99% of trials should be written on this label (no sec style everything is a risk word vomit disclosures). There should not be a pre-emptive application for a label - it can only be gotten after paper submission to reduce gaming. There should be an independent advisory org that scientists can literally call to ask for advice on structuring the trials. These calls must not be disclosed. Much like farmers can call the government to ask for help on xyz crop problem. And these labels should never be used as the primary source of punishment. Any and all sanctions/penalties/dismissals must go through a new review process done by a different group. Any scientist who gets a label in a particular year should be given a vote to review the review agency on several dimensions. These aggregate reviews should be published broadly but not trigger any automatic consequences. Clear, accessible information is the basis for any self regulating human system. We need more of it in this field.
- alphazard 3y ago
  The success of the Nutrition Facts labeling does not get enough publicity. Rather than outlawing certain ingredients, or creating some kind of health score which a product must be above, Nutrition Facts is a way for suppliers to attest to information about a product in way that is legally binding. If it isn't accurate the penalties are steep. Consumers then have the information they need to vote with their wallets. Markets cannot function properly without symmetric information, and Nutrition Facts essentially creates a functioning market where one did not exist previously. Any effort to regulate what's in food would probably be better spent expanding what must be in the Nutrition Facts label. I guess it's nice that we are finally getting around to banning artificial trans fats, but anyone who can read has been able to keep those out of their diet for years. The same can be said about the next bad ingredient, and the one after that.
light_hue_1 3y ago
This is a problem everywhere where the raw data isn't released (suitably anonymized). In cognitive science, psychology, even computer science / ai / ml, business. And the problem with rejected papers getting in somewhere else while being total garbage is pervasive. I've rejected a lot of papers because they were mathematically or statistically bogus only to see them get published elsewhere where reviewers were not so careful (a few times in Nature and Science). We need an open science movement where you must release everything with your paper. The full pipeline to reconstruct every single result from the raw data. No hiding data. No hiding fmri scans. No "our code only runs on our machine". Etc.
- v4dok 3y ago
  I think value-based care is the only real incentive on this. Otherwise, there is simply no reason for anyone to care enough. Even the insurers, they found a way to make money by making sure their premiums factor in these things. In the expense of the patient. As long as the drug doesn't kill people, who cares if it works if I make money off it as a pharma? Unfortunately, value-based care can only be pushed top-down. Patients are not in a position of power against pharma companies on this matter.
- derbOac 3y ago
  I'm empathetic to what you're suggesting — I've published on open science and meta-science specifically, and think open data should be the default norm. The problem with clinical research, though, is that it starts running into conflicting considerations about patient and participant privacy. Even when people aren't patients per se, the focus often involves sensitive information. You can just say "anonymize it" but that turns out to be more difficult than it seems initially, especially with many questions of interest. Also, there's often too many opportunities to do science that is of real public benefit that comes with privacy expectations attached for all kinds of reasons. Cases where there is legitimate consented access but an expectation of privacy without data sharing. People have tried to solve this problem in different ways (for example, methods where someone can analyze data without having access to it directly) and maybe those solutions will lead to a good resolution. But they often have problems of their own (overhead costs associated with providing anonymized remote data analysis), and don't solve all problems (guarantees of absolutely restricted access to personal data).
egberts1 3y ago
I now know of a few more people who have completely lost both of their hearing after taking just one Wellburtin-class pill. At the time, no mention was made in the pill’s warning pamphlet. It is still difficult to secure a class-action suit in America. Meanwhile, such quality of life would plummet into a silent world, even if one knew American Sign Language fluently beforehand, that tidbit can go against the victim in court.
appleflaxen 3y ago
If you read the article, the headline is wildly editorialized. Whatever. Par for the course in 2023, right? But this is Nature, a paragon of scientific literature, fueling the distrust of medicine. There are great reasons to be skeptical of all trials and strengthen peer review and transparency, but this kind of headline is editorial malpractice, in my opinion.
- bmh 3y ago
  [flagged]
  - defrost 3y ago
    [flagged]
    - dablweb 3y ago
      [flagged]
    - bmh 3y ago
      [flagged]
  - swader999 3y ago
    [flagged]
  - LatteLazy 3y ago
    [flagged]
  - ekianjo 3y ago
    Closer to 50% of trials are probably junk since they cant be reproduced independently
    - throw9away6 3y ago
      That’s being generous as only 1/10 can actually be fully reproduced from my understanding. It’s so bad that if you want to create a product based on research your first need to reproduce the result to make sure it’s not bs.
mjfl 3y ago
It’s hard enough to run a clinical trial guys. It literally costs $100 million at a minimum, yet the requirements that make it cost this much are not enough. We are basically going to regulate new medicine out of existence.
- WastingMyTime89 3y ago
  Please read the article before commenting. The problem is not how hard it is to run clinical trial. It's that made up data is an endemic problem. It doesn't matter if clinical trials are hard or easy to organise when up to a quarter don't actually bother and just forge their results.
  - chaxor 3y ago
    But they have to make up that data! Because the work they are doing now was based on a other trial where they made up data, so you have to fix this data to match what was expected from the previous studies. Of course we need to protect their right to make up data. /s
- droopyEyelids 3y ago
  Hamilton Morris recently did a podcast that touched on some of this, I think it was https://www.patreon.com/posts/pod-78-legal-84786504 https://www.patreon.com/posts/pod-78-legal-84786504 An interesting point he made was that in the wake of the Thalidomide (https://en.wikipedia.org/wiki/Thalidomide https://en.wikipedia.org/wiki/Thalidomide) scandal, the FDA started requiring drugs to be both safe _and effective_ https://www.fda.gov/about-fda/histories-product-regulation/promoting-safe-effective-drugs-100-years https://www.fda.gov/about-fda/histories-product-regulation/p... The 'effective' part has proven to be a big source of complexity in the following years, because while it's relatively easy to prove a drug is relatively safe, it's much more difficult and subjective to prove a drug is effective. That closes off a lot of areas of research and development. The kicker is that Thalidomide was never sold in the USA to begin with. Anyway, as any reader can imagine, there would be a lot of negative social outcomes to allowing the sale of ineffective drugs. There's a lot of trouble now with medical devices and drugs not being effective, even though we have the rule. I'm not against regulation, I think medical sales are a really complex issue and I don't know how to even judge where the right balance of safety/effectiveness and innovation/freedom could be.
alexb_ 3y ago
I'm not too knowledgeable when it comes to how scientific experiments/trials are done - are the people who collect the data, the people who interpret the data, and the people who fund/benefit from the data different parties? Or are they the same people?
- WaitWaitWha 3y ago
  In my experience, they are different (speaking as someone witnessing it from layman's perspective). Here is what I have seen in Phase II & III trials: a) Pharma identities the type of patients they need (e.g., 25-50 female, not pregnant, with specific ailment if Phase III), specific tests, and measurements required throughout the study. b) pharma contacts third party (3P) to manage study patients. c) 3P has relationship with dozens or even hundreds of doctors' offices, knowing what office can fulfill the test & measure requirement, and has the potential trial patient pool. d) 3P has existing contract with these doctors' & hospitals. They get patients onto the study. ( <--- #1 reason this is farmed out in my opinion) e) Doctors & hospitals perform the study and collect the data. f) doctors & hospitals pass the data to the 3P g) 3P passes it to the pharma h) repeat e) through g) as many times as the study requires it. This can be once, or many times over years. i) pharma pays 3P, 3P pays doctors & hospitals, and they pay the trial patients - each taking their cut along the way. There are variations on how this is done, sometimes no 3P, sometimes pharma will have their own pool of patients and 3P. Also this is a very rough flow as there are often checks, audits, and validations (should be) done during the study.
- linuxftw 3y ago
  For vaccine manufacturers, it's all the same people. Even if you have whistleblowers come forward, they're ignored. Here's one quick way to rig any clinical trial: Anyone and everyone that has any kind of negative reaction or health condition gets disenrolled. Since it's 'double blind' it appears on the surface level that there's no way to know who's in what group. Naturally, the end result is always the same: The test group had the same number of reactions as the control by the end of the study.
  - nocoolnametom 3y ago
epicEHRsucks 3y ago
There unfortunately are too many perverse incentives that encourage fraudulent studies. Everyone should take findings "established in the literature" with a grain of salt. We should also incorporate more intuition and first-principles reasoning, i.e. obesity is a harmful state for humans even if 100 RCTs proved otherwise.
constantcrying 3y ago
How can you do science if around a quarter of the data is just straight up noise? Even when analysing large amounts of studies the results becone contaminated very easily. And why is it not standard practice to provide anonymous data or even publish the data? What reason exists for that? So that only the researchers them selves can analyze it?
- thechao 3y ago
  The working stuff becomes part of a body of lore of "real science" that you soak up in the lab. Not a great method, for sure.
  - constantcrying 3y ago
    But the point of RCTs is that you can get unbiased, high quality results without having to rely on "lore" spreading among medical professionals about which treatments are effective and under which circumstances.
- mike_hearn 3y ago
  Yes the problem is that the research system rewards publishing papers, but most of the work is collecting the data. So if you release your data then other groups can write papers based on your effort for far less cost. It's sort of analogous to the problem of open source business models in the software world: if company A writes the code and releases it for free and earns money from running a cloud service, and company B just offers a cloud service, then the second company can get much higher margins because they don't have to develop it. Unfortunately it's not easy to see what the alternatives are, beyond simply not funding research through government/foundation grants. When science is paid for by companies you don't have this incentive issue because the research is judged based on (ultimately) whether it leads to successful products, not whether it leads to lots of papers getting published. You have other incentive issues of course.
- twic 3y ago
  That's actually mentioned in the article, which is quite good and worth reading: > In 2016, the International Committee of Medical Journal Editors (ICMJE), an influential body that sets policy for many major medical titles, had proposed requiring mandatory data-sharing from RCTs. But it got pushback — including over perceived risks to the privacy of trial participants who might not have consented to their data being shared, and the availability of resources for archiving the data. As a result, in the latest update to its guidance, in 2017, it settled for merely encouraging data sharing and requiring statements about whether and where data would be shared.
oldgradstudent 3y ago
With the risk of starting a flame war, we recently had several well-publicised clinical trials that reported 95% efficacy of some certain modality. Yet, in reality, efficacy as defined in the trial turned out to be closer to 0%. Instead of investigating what in the design and execution of the trial led to such a discrepancy, the problem was handled by denying there was a problem, changing the goalposts, reporting ad-hoc hypotheses as facts, silencing all critics, and forcing the public to take the modality anyway or lose jobs, school, and freedom of movement.
- taeric 3y ago
  I can only guess you mean something regarding the pandemic? Do you have links to show things were "closer to 0%?" Sounds more than a touch outlandish. :( I'm confident we will know even more about things as time goes on. I'm less confident on any nefarious motivations in most of it. Reality is that a lot of people died, and everyone was trying to gain control and an advantage over the situation. Mistakes were certainly made, but I am back to low confidence in thinking that everything was a mistake.
  - oldgradstudent 3y ago
    >Do you have links to show things were "closer to 0%?" Sounds more than a touch outlandish. :( Take the Pfizer vaccine. The clinical trial's main endpoint was ~95% efficacy in one thing and one thing only, prevention of symptomatic Covid. Not reduction in mortality, not reduction in serious disease, not infection, and not spread. The only thing the trial tested and reported was prevention of symptomatic Covid. This is also the sole indication in the package insert as approved by the FDA. In reality, everyone I know got vaccinated and got symptomatic Covd. I mean everyone, no exceptions. The situation is the similar in my entire country and around the world.
    - taeric 3y ago
      Do you know of any studies on the discrepancy? My understanding was that Omicron came out and basically gave the middle finger to everyone's precautions. With what seemed like literally nothing working against it.
mydriasis 3y ago
More and more it seems that science is being clouded by moneyed interests and greed. If we can't trust science, what can we trust?
- thechao 3y ago
  This is just reporting bias. I've worked (on-and-off) in various scientific fields for ~30; it has always had its bad actors. (I even helped do the statistics for some!) I'd urge you to consider following the situations: 1. Prescientific inquiry; and, 2. PreFDA food, drug, and medicine. Both of those were orders-of-magnitude worse than what we have, now. Could we do better? Sure! Is it broken? No.
  - mydriasis 3y ago
    That brings a bit of hope!
  - LorenPechtel 3y ago
    Disagree. Broken isn't a binary--the current system is far better than what came before, but that doesn't mean there aren't serious flaws in the current system.
- jasmer 3y ago
  [dead]
gigatexal 3y ago
In the age of QAnon and Alex Jones and science and vaccine deniers we can’t have things like clinical studies be corrupted. Ugh.
- sonicshadow 3y ago
  Yeah not a great look, maybe science isn’t the truth after all
- linuxftw 3y ago
  The 'vaccine deniers' have been calling the 'studies' and 'trials' trash for a long time. The evidence comes out that they're trash, and somehow the 'vaccine deniers' are still wrong? They removed liability from manufacturers, and suddenly the CDC schedule exploded with new products. I mean, chicken pox has a vaccine now?
  - nocoolnametom 3y ago
    That which was asserted without evidence could correctly be dismissed without evidence. AT THE TIME the anti-vax crowd was basing their positions entirely upon anecdote, rumor, and often badly misread prepublication research and stats. Their methodology was inherently flawed. Even if the conclusions they came to have been "validated" their position was still built upon this same flimsy scaffolding. It's not like the "do your own research" blogs and videos somehow gathered the same evidence used by this paper. This also does not indicate that other positions held by the same crowd, which are similarly based upon "anecdata" and rumor, are somehow made more evident by this paper in Nature.
    - linuxftw 3y ago
      I agree, the assertion that 'vaccines' are safe can be dismissed without evidence. There's no evidence concluding they're actually safe. In fact, we have given the manufacturers immunity because they're 'unavoidably unsafe.' Just look at how the COVID trials were conducted. They didn't even test each patient. Only some patients that presented symptoms, and then not even all of those patients. How long did they follow the health outcomes for approved vaccines in the test groups? 3 months at most, and many trials, not even that long. So if someone suffers a neurological condition, well, we just won't know about it.
dablweb 3y ago
Ironic that nature publishes this despite being guilty of constantly promoting it.
jmpeax 3y ago
Raw data examined: Ok 56%, Flawed 18%, Zombie 26% Raw data not available: Ok 97%, Flawed 2%, Zombie 1% Perhaps it's not good to call an unknown as "Ok"? Maybe Carlisle should add his own paper to the mix?
- twic 3y ago
  That's literally the point he's making! The terms "flawed" and "zombie" reflect positive identification of dodgy data, so of course when the data isn't available, they are less likely to apply, hence why: > This finding alarmed him, too: it suggested that, without access to the IPD — which journal editors usually don’t request and reviewers don’t see — even an experienced sleuth cannot spot hidden flaws.
deleted 3y ago
[deleted]
gumby 3y ago
Note that the article discusses research studies and not clinical trials for drug or device approval. These research studies are important (look at how many were conducted on COVID-19 over the last few years) but are typically not held to a particularly high standard, as with most science. Which doesn’t excuse bad data or poor statistics (the latter supposedly supposed to be picked up in peer review).
- piqufoh 3y ago
  Hmm, I read the article as explicitly calling out "clinical trials" (as referenced in the title and abstract) and it makes no reference research studies. I don't understand the distinction between "research studies" and "clinical trials", surely all research studies where an RCT is performed with real patients and real drugs is a clinical trial?
  - gumby 3y ago
    I meant “trials for research studies” as opposed to “trials for drug or device approval.” The amount of record keeping and oversight of a drug approval trial is enormous (and as a consequence insanely expensive) — data handling, having disjoint groups at each stage handling and analyzing data, etc and detailed records of every manufacturing step — think ISO9000 on steroids. Nobody would bother to go to that effort for a scientific exploration, nor should they. So the bar is much lower. I am making no excuse for shoddy science! But it is quite unlikely for a licensed drug.
freedude 3y ago
“I think journals should assume that all submitted papers are potentially flawed and editors should review individual patient data before publishing randomised controlled trials,” Carlisle wrote in his report. This should be considered part of every journal's idea of due diligence and this shouldn't be a new idea. Shysters, con artists and snake oil salesmen have been around for a long time. The purpose of a Journal is to publish reliable information and weed out the garbage. How can you do that if you are not looking at the entire picture?
isaacremuant 3y ago
No. no. This is not permitted. Trust the science or you're an antivaxer, Trumper, denier, racist, white supremacist. Media and gov, or media at the behest of gov (as the twitter files prove, but it was obvious without them), censored as much as thet could, everything that went against what they wanted to push. It wasn't science but security/hygiene theater and it worked, because people did go along and did turn on their neighbours who opposed the measures. But now we slowly get tidbits of things we can debate again ... Funny that, but no recognition that the entire lockdown, masks and vaccine mandate effort + economic destruction and theft (tax money to corps as "aid") was never a reasonable, logical or scientific response. It was an authoritarian and corrupt response of extreme Effectiveness and cynicism. But here comes some guy to say "people have always quarantined in pandemics" or some other disingenuous claim that ignores the reality of what happened: healthy people denied basic human and constitutional rights.
- LorenPechtel 3y ago
  The problem here is "healthy people". Covid's real key to success is the fact that most spread is presymptomatic. It's the apparently healthy people spreading it! And note the lessons of history: 1) There will always be those who choose short term economic interests over safety when the threat isn't absolutely proven. They'll always close the barn door too late. 2) Places that take epidemic/pandemic threats seriously tend to fare better economically in the long run.
  - isaacremuant 3y ago
    No. The problem is that fundamentally, you and people like you decided that authortiarianism is ok if the gov says "things are scary" and everyone should suffer the consequences. No debate allowed. The risk profile for people was always ridículous and you were never going to contain it once it was widespread, which it was, but somehow we believed in the rolling "it's just 2 weeks". 1) it wasn't short term. It wasn't safety. Short and long term you hurt the vulnerable and you helped the rich and powerful. 2) Rich places will keep being richer and poor places poorer, and when the powers that be decided that enough was enough, all the concerns of the hypochondriacs suddenly didn't matter. You could tell, if you paid attention, that politicians weren't afraid after the initial surge, but wanted "the masses" to be. You even have definite proof in many places, one of them being UK and number 10. Can't link it now, sorry. But Google downing Street covid rules or something on that note and you can probably find a lot. It wasn't limited to the UK. It was everywhere you looked properly. You were scammed and you either were well off and didn't mind that much or you want to pretend you weren't for your own mental health. Because the truth is a hard pill to swallow.
xhkkffbf 3y ago
Even if they aren't faked, the ability to shut down a trial that isn't delivering the right preliminary data is a big problem for society. Why should we trust these drugs?
- throw9away6 3y ago
  A lot of these studies are prerun in the 3rd world before the real one is done for credit. If sideffects show up the drug can sometimes be mixed with one that has the same known side effects to fool the studies.
Roark66 3y ago
They say "medicine". I would say science in general, perhaps we could generalise even further to "any human activity is full of unethical people trying to exploit it". But 25%!? That suggest there is a big problem with how we "do science". Unfortunately I have no solution to the problem. Publishing everything (including raw data) for every research would probably help somewhat, but only teams repeating experiments/trials would ensure it. I wonder if we suddenly took 10% of all money spent on science (let's say in medicine) and instead of novel research we used it to redo randomly chosen previous research. Would we loose or gain in terms of new cures? And if we gained, what if we spent 15%, or 25%? That's a great idea for a scientific study to find a point of diminishing returns on "research verification". Would someone please write a research grant request for this?
- davidktr 3y ago
  Of course there are big problems with how we do science. Much of it is garbage. Imagine most software was written by junior engineers, without any code review or input from seniors. That is today's science. Most scientific legwork is done by absolute beginners, i.e., graduate students. They often lack a support structure to focus on what they have learned so far. Most of the world is not Oxbridge, MIT, Stanford. Where are the beginners' supervisors, you might ask? Chasing the latest trend to secure funding. Pondering how their line of research can be formulated as buzzword-du-jour markov chain. Ass kissing the dean to get department funding. Having worked in research for 15 years, I am certain about two things: (1) The scientific method yields better results than doing things freehand. (2) Randomly axing 50% of academia would improve the situation.
- bobbylarrybobby 3y ago
  That 10% investment would have a huge payoff too because it would shut down avenues that were only opened due to p-hacking long before they'd had a chance to seek further investment.
deleted 3y ago
[deleted]
j-pb 3y ago
That and lawyer driven development. I have multiple sclerosis, and I never know if a new medication came out because it's actually better or simply because they could get a new patent for a slightly modified molecule.
- LorenPechtel 3y ago
  Disagree--you're talking about patent-driven development, not lawyer-driven development.
thenerdhead 3y ago
Another good reason why you should learn how to read studies: https://peterattiamd.com/ns001/ https://peterattiamd.com/ns001/ https://biolayne.com/reps/how-to-read-research-a-biolayne-guide/ https://biolayne.com/reps/how-to-read-research-a-biolayne-gu... https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7392212/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7392212/ And as a bonus, read books by controversial figures who talk about these challenges through their own published studies and decided it isn't worth fighting. What would be cool is to see "trust indicators" as part of a study's metadata as it is reviewed through time and continually shared by others. Could be a "study health score" or a checklist showing the study isn't biased heavily by sponsors, methods, or misleading language. Especially given that medicine is becoming more personal with the advancements of AI and accessibility of tests, someone should be able to understand the health of a study at-a-glance reviewed by peer reviews rather than disclosed only by the authors.
- randcraw 3y ago
  A primary trust indicator of any research result is the reputation of the journal publishing the paper. Better journals demand more, not just in terms of the impact/insight of the results, but also the rigor of the experimental methodology used and how well the data was curated and the confounding variables identified and isolated. And of course, the criteria for observational-based research differ a lot from mechanism-based, especially since the former can't control for nearly as many variables. The same goes for simulations or interpretive modeling, where experimentation is minimal. Research studies differ quite a lot in how precisely the mechanism of action is identified, isolated, tested, and results interpreted. IMHO, blackening the trustworthiness of all of science is unhelpful, especially when some models and methods are surely more trustworthy and replicable than others. Better to identify and catalog specific sources of error (or imprecision) in order to remedy them than to just toss the baby.
darkclouds 3y ago
I'd trust the NHS if the National Institute of Clinical Excellence (NICE) actually published their minutes online. Most employees only know what they have been taught, so whilst I agree with the headline that some trial data is dodgy, it doesnt just end there. There are multiple pathways to factor in, there is redundancy built in, ie secondary pathways, there is the fact that not all chemicals go where intended (best highlighted by radioactive isotopes). When looking at the history of patented medicine, this really started between WW1 and WW2, before WW1, most GP's prescribed what was found in the body on a like for like basis, and in some cases prescribed organs in various forms, like desiccated thyroid gland for thyroid related problems, for pernicious anaemia, raw liver used to be prescribed to women and so on, but that has its problems like contamination and diseases. So from WW1/WW2 onwards the rise of patented medicine took hold, but the main problem with patented medicine is the human body hasnt evolved to use these new chemical compounds in the same way as unpatented chemicals which have been around for thousands of years. And todays GP's dont really highlight the side effects of the patented medicine, and because they simply dont ask what you have been eating and drinking etc, they have this hubris which sucker punches your trust and sucks you in like a black hole, until before you know it you are on half a dozen different drugs, your quality of life is going down the pan and you've been left a zombie wondering where did Hitler go wrong modelling the German state of the time on the British Empire? You Americans complaining about the cost of healthcare, should count yourself lucky those insurance companies are looking out for your long term interests and theirs! Fortunately, hospitals in other countries publishes studies and as english seems to be the main language used for science, I have to tip my hat to the Chinese who are roaring up the charts in terms of investigating and publishing relevant studies that will complement a quality of life one hopes to achieve, and we cant forget Wikipedia, Pubmed and Google for connecting users with pertinent studies. Saying that I do sometimes wonder if something like ChatGPT has written some studies due to the poor quality of english used, but generally they stand out like a sore thumb. Anyway, does any know why there is a connection between MDMA, blood clotting and Manganese?
amai 3y ago
There should be a paperswithdata for medicine like there is a https://paperswithcode.com/ https://paperswithcode.com/ for data science.
jet_32951 3y ago
Have a look at Derek Lowe's excellent blog [0] to view the depths of malfeasance, if not outright fraud, in which "studies" are created. [0] https://www.science.org/content/blog-post/fakin-it-modern-way https://www.science.org/content/blog-post/fakin-it-modern-wa... is just his latest in a long string of well-documented posts.
OnlyMortal 3y ago
To be fair, science is now a “publish or be damned” business. So many papers are not worthy of publication and, frankly, are of common knowledge anyway. Of course, this is driven by the money backing the research which, I’m sure, this is why clinical trials in the medical areas can be of a poor quality. People want to keep their jobs.
tails4e 3y ago
Nutrition for me it a big issue. It seems like and ideal area for scientific and medical study to give us light, but seems so hard to get truly objective info. Fat is the devial/good/bad/OK, sugar is the devil/bad/ok, etc. Should I avoid white bread like a hole in the head, or is it fine? Too many agendas and not enough truth.
jononomo 3y ago
I don't think it makes sense to publish the results of clinical trials and other scientific experiments until after they have been independently replicated.
Madmallard 3y ago
This is entropy in action. We're not going to move forward as long as malicious behavior is rewarded with money in large-scale systems. I suspect in coming decades more incurable chronic illness will become rampant and polypharmacy will become more common and necessary for larger percentages of the population. It is really sad to see.
dennis_jeeves1 3y ago
I will start my own medical 'system' and be my own doctor. To hell with the mainstream and it's corrupt/stupid adherents.
pyuser583 3y ago
It doesn’t help that front-line doctors are expected to publish.
NoZebra120vClip 3y ago
It was interesting to find out that test subjects who pass away during clinical trials are excluded, erased, from the trial records. Their participation and data is not included in the final results, as if they were never in the trial to begin with. Many pharmaceutical companies have policies which prohibit patients from participating in studies for anti-depressants if they have suicidal ideations. In practice, this significantly reduces the participation of any patient with depression.

333 points by headalgorithm 3y ago

Masters and Phd requirements should include repeating another research and checking whether they can recreate the results

sealeck 3y ago
Unfortunately some of the experiments require a whole team to carry out and take a huge amount of time to set up as well as having to be carried out under very specific conditions.
Slaminerag 3y ago
My institution's been included in studies primarily for patient access. If we're not included, then good luck getting enough patients, and even then it can take several years to enroll enough patients. Replicating such a trial would be near impossible.

__666__ 3y ago

[flagged]

wahnfrieden 3y ago
Is it structural greed?

CrampusDestrus 3y ago

it's 2023, we have the means to cheaply record and store audio and video evidence for basically any medical experiment. we can record every patient reaction and opinion without relying on the reasearchers' hearsay. we also have the means to store and distribute all the binary/textual raw data gathered throughout the experiments. maybe as an intermediate step we could make available all the recordings to the peer reviewers and only offer the raw experimental data bundled in the paper publicly? maybe in the future we can have 1TB studies without breaking a sweat? maybe all the money we give to publishers can be spent on servers to archive all the primary data so at least we aren't simply filling the pockets of MBAs?

jamesdwilson 3y ago
how do you prevent cherry picking?
- CrampusDestrus 3y ago
  that's such a vague question for example, if you have 50 partecipants but only provide the multimedia evidence for 20 of them your study should be thrown out the window
  - dolni 3y ago
    What is preventing someone from having 200 participants, but saying they only had 100 participants, and then only providing evidence for 100 participants?
    - CrampusDestrus 3y ago
      The researchers will have to have gone through some kind of third party agency to get the partecipants. This agency should be queried to see the number they report
      - dolni 3y ago
        How does this agency determine who can meaningfully participate in the study? Are they going to have the expertise to make that determination for _every_ study that could possibly be conducted? What is the difference (to a layperson) between cherry-picking participants and rejecting participants because they do not meet your study's criteria? Who funds this agency? Do the members of the reviewing agency have their own biases, and might those biases tarnish the reputation of a study that is actually well-conducted? (hint: this already happens in journals)
    - throw9away6 3y ago
      You have to declare the study population before you request the next round of funding. Thereby fixing the problem.

Eumenes 3y ago

The over prescription of statins is a great example of what pharma sponsored clinical trials result in

remote_phone 3y ago
This. Statins claimed to be free from adverse reactions, but it turns out that about 30% of the participants were taken out of their clinical study because of “non-compliance”. However, if you dig in further, the non-compliance was because of adverse reactions. You can’t trust pharma companies if their data is secret.
- TheBigSalad 3y ago
  There must be more to it. Everyone who drops out is scrutinized.
- anonuser123456 3y ago
  If you dig further into those adverse reactions, you’ll find they are approximately equal to adverse reactions of placebo.
  - Madmallard 3y ago
    Except for the part where there is a clear mechanism and cause related to muscle damage and cellular dysfunction.
    - anonuser123456 3y ago
      Show me the double blinded, placebo controlled study.
      - Madmallard 3y ago
        https://www.acc.org/about-acc/press-releases/2016/04/03/15/02/trial-offers-objective-evidence-of-muscle-related-side-effects-with-statins https://www.acc.org/about-acc/press-releases/2016/04/03/15/0... here you go anonuser123456 You will likely be able to find scientific studies toward any conclusion you want to make but there is objective evidence of this and in my own personal life significant anecdotal evidence. Given the OP title and topic and the massive reproducibility problem in medicine and perverse incentive problem in medicine, you will not be able to know the truth about the safety of medications to a high degree. It is like the reviews problem on Amazon. All medications come with studies giving 5 star reviews. But the reality is nowhere near the same. Blackbox warnings and such are all we have to place a significant ding on this problem before taking the drugs off the market for clearly causing intolerable issues in large swaths of people. This is a complex muddled issue. Argumentation with citations of 'articles' from journals likely will not lead to truthful discovery, unfortunately.

obblekk 3y ago

There should be Nutrition Facts but for scientific trials. Independent agency just publishing quality assessments of the trial. This should be an async non blocking evaluation. The statisticians who do it should be anonymous by default. There should be an appeals process for a scientist to explain why an unconventional new method is actually robust. There should not be a single number published by this process, but rather a list of stats that speak to the overall quality of the trial on many dimensions (power, sources of bias, etc). Only information that would not be the same on 99% of trials should be written on this label (no sec style everything is a risk word vomit disclosures). There should not be a pre-emptive application for a label - it can only be gotten after paper submission to reduce gaming. There should be an independent advisory org that scientists can literally call to ask for advice on structuring the trials. These calls must not be disclosed. Much like farmers can call the government to ask for help on xyz crop problem. And these labels should never be used as the primary source of punishment. Any and all sanctions/penalties/dismissals must go through a new review process done by a different group. Any scientist who gets a label in a particular year should be given a vote to review the review agency on several dimensions. These aggregate reviews should be published broadly but not trigger any automatic consequences. Clear, accessible information is the basis for any self regulating human system. We need more of it in this field.

alphazard 3y ago
The success of the Nutrition Facts labeling does not get enough publicity. Rather than outlawing certain ingredients, or creating some kind of health score which a product must be above, Nutrition Facts is a way for suppliers to attest to information about a product in way that is legally binding. If it isn't accurate the penalties are steep. Consumers then have the information they need to vote with their wallets. Markets cannot function properly without symmetric information, and Nutrition Facts essentially creates a functioning market where one did not exist previously. Any effort to regulate what's in food would probably be better spent expanding what must be in the Nutrition Facts label. I guess it's nice that we are finally getting around to banning artificial trans fats, but anyone who can read has been able to keep those out of their diet for years. The same can be said about the next bad ingredient, and the one after that.

light_hue_1 3y ago

This is a problem everywhere where the raw data isn't released (suitably anonymized). In cognitive science, psychology, even computer science / ai / ml, business. And the problem with rejected papers getting in somewhere else while being total garbage is pervasive. I've rejected a lot of papers because they were mathematically or statistically bogus only to see them get published elsewhere where reviewers were not so careful (a few times in Nature and Science). We need an open science movement where you must release everything with your paper. The full pipeline to reconstruct every single result from the raw data. No hiding data. No hiding fmri scans. No "our code only runs on our machine". Etc.

v4dok 3y ago
I think value-based care is the only real incentive on this. Otherwise, there is simply no reason for anyone to care enough. Even the insurers, they found a way to make money by making sure their premiums factor in these things. In the expense of the patient. As long as the drug doesn't kill people, who cares if it works if I make money off it as a pharma? Unfortunately, value-based care can only be pushed top-down. Patients are not in a position of power against pharma companies on this matter.
derbOac 3y ago
I'm empathetic to what you're suggesting — I've published on open science and meta-science specifically, and think open data should be the default norm. The problem with clinical research, though, is that it starts running into conflicting considerations about patient and participant privacy. Even when people aren't patients per se, the focus often involves sensitive information. You can just say "anonymize it" but that turns out to be more difficult than it seems initially, especially with many questions of interest. Also, there's often too many opportunities to do science that is of real public benefit that comes with privacy expectations attached for all kinds of reasons. Cases where there is legitimate consented access but an expectation of privacy without data sharing. People have tried to solve this problem in different ways (for example, methods where someone can analyze data without having access to it directly) and maybe those solutions will lead to a good resolution. But they often have problems of their own (overhead costs associated with providing anonymized remote data analysis), and don't solve all problems (guarantees of absolutely restricted access to personal data).

egberts1 3y ago

I now know of a few more people who have completely lost both of their hearing after taking just one Wellburtin-class pill. At the time, no mention was made in the pill’s warning pamphlet. It is still difficult to secure a class-action suit in America. Meanwhile, such quality of life would plummet into a silent world, even if one knew American Sign Language fluently beforehand, that tidbit can go against the victim in court.

appleflaxen 3y ago

If you read the article, the headline is wildly editorialized. Whatever. Par for the course in 2023, right? But this is Nature, a paragon of scientific literature, fueling the distrust of medicine. There are great reasons to be skeptical of all trials and strengthen peer review and transparency, but this kind of headline is editorial malpractice, in my opinion.

bmh 3y ago
[flagged]
- defrost 3y ago
  [flagged]
  - dablweb 3y ago
    [flagged]
  - bmh 3y ago
    [flagged]
- swader999 3y ago
  [flagged]
- LatteLazy 3y ago
  [flagged]
- ekianjo 3y ago
  Closer to 50% of trials are probably junk since they cant be reproduced independently
  - throw9away6 3y ago
    That’s being generous as only 1/10 can actually be fully reproduced from my understanding. It’s so bad that if you want to create a product based on research your first need to reproduce the result to make sure it’s not bs.

mjfl 3y ago

It’s hard enough to run a clinical trial guys. It literally costs $100 million at a minimum, yet the requirements that make it cost this much are not enough. We are basically going to regulate new medicine out of existence.

WastingMyTime89 3y ago
Please read the article before commenting. The problem is not how hard it is to run clinical trial. It's that made up data is an endemic problem. It doesn't matter if clinical trials are hard or easy to organise when up to a quarter don't actually bother and just forge their results.
- chaxor 3y ago
  But they have to make up that data! Because the work they are doing now was based on a other trial where they made up data, so you have to fix this data to match what was expected from the previous studies. Of course we need to protect their right to make up data. /s
droopyEyelids 3y ago
Hamilton Morris recently did a podcast that touched on some of this, I think it was https://www.patreon.com/posts/pod-78-legal-84786504 https://www.patreon.com/posts/pod-78-legal-84786504 An interesting point he made was that in the wake of the Thalidomide (https://en.wikipedia.org/wiki/Thalidomide https://en.wikipedia.org/wiki/Thalidomide) scandal, the FDA started requiring drugs to be both safe _and effective_ https://www.fda.gov/about-fda/histories-product-regulation/promoting-safe-effective-drugs-100-years https://www.fda.gov/about-fda/histories-product-regulation/p... The 'effective' part has proven to be a big source of complexity in the following years, because while it's relatively easy to prove a drug is relatively safe, it's much more difficult and subjective to prove a drug is effective. That closes off a lot of areas of research and development. The kicker is that Thalidomide was never sold in the USA to begin with. Anyway, as any reader can imagine, there would be a lot of negative social outcomes to allowing the sale of ineffective drugs. There's a lot of trouble now with medical devices and drugs not being effective, even though we have the rule. I'm not against regulation, I think medical sales are a really complex issue and I don't know how to even judge where the right balance of safety/effectiveness and innovation/freedom could be.

alexb_ 3y ago

I'm not too knowledgeable when it comes to how scientific experiments/trials are done - are the people who collect the data, the people who interpret the data, and the people who fund/benefit from the data different parties? Or are they the same people?

WaitWaitWha 3y ago
In my experience, they are different (speaking as someone witnessing it from layman's perspective). Here is what I have seen in Phase II & III trials: a) Pharma identities the type of patients they need (e.g., 25-50 female, not pregnant, with specific ailment if Phase III), specific tests, and measurements required throughout the study. b) pharma contacts third party (3P) to manage study patients. c) 3P has relationship with dozens or even hundreds of doctors' offices, knowing what office can fulfill the test & measure requirement, and has the potential trial patient pool. d) 3P has existing contract with these doctors' & hospitals. They get patients onto the study. ( <--- #1 reason this is farmed out in my opinion) e) Doctors & hospitals perform the study and collect the data. f) doctors & hospitals pass the data to the 3P g) 3P passes it to the pharma h) repeat e) through g) as many times as the study requires it. This can be once, or many times over years. i) pharma pays 3P, 3P pays doctors & hospitals, and they pay the trial patients - each taking their cut along the way. There are variations on how this is done, sometimes no 3P, sometimes pharma will have their own pool of patients and 3P. Also this is a very rough flow as there are often checks, audits, and validations (should be) done during the study.
linuxftw 3y ago
For vaccine manufacturers, it's all the same people. Even if you have whistleblowers come forward, they're ignored. Here's one quick way to rig any clinical trial: Anyone and everyone that has any kind of negative reaction or health condition gets disenrolled. Since it's 'double blind' it appears on the surface level that there's no way to know who's in what group. Naturally, the end result is always the same: The test group had the same number of reactions as the control by the end of the study.
- nocoolnametom 3y ago

epicEHRsucks 3y ago

There unfortunately are too many perverse incentives that encourage fraudulent studies. Everyone should take findings "established in the literature" with a grain of salt. We should also incorporate more intuition and first-principles reasoning, i.e. obesity is a harmful state for humans even if 100 RCTs proved otherwise.

constantcrying 3y ago

How can you do science if around a quarter of the data is just straight up noise? Even when analysing large amounts of studies the results becone contaminated very easily. And why is it not standard practice to provide anonymous data or even publish the data? What reason exists for that? So that only the researchers them selves can analyze it?

thechao 3y ago
The working stuff becomes part of a body of lore of "real science" that you soak up in the lab. Not a great method, for sure.
- constantcrying 3y ago
  But the point of RCTs is that you can get unbiased, high quality results without having to rely on "lore" spreading among medical professionals about which treatments are effective and under which circumstances.
mike_hearn 3y ago
Yes the problem is that the research system rewards publishing papers, but most of the work is collecting the data. So if you release your data then other groups can write papers based on your effort for far less cost. It's sort of analogous to the problem of open source business models in the software world: if company A writes the code and releases it for free and earns money from running a cloud service, and company B just offers a cloud service, then the second company can get much higher margins because they don't have to develop it. Unfortunately it's not easy to see what the alternatives are, beyond simply not funding research through government/foundation grants. When science is paid for by companies you don't have this incentive issue because the research is judged based on (ultimately) whether it leads to successful products, not whether it leads to lots of papers getting published. You have other incentive issues of course.
twic 3y ago
That's actually mentioned in the article, which is quite good and worth reading: > In 2016, the International Committee of Medical Journal Editors (ICMJE), an influential body that sets policy for many major medical titles, had proposed requiring mandatory data-sharing from RCTs. But it got pushback — including over perceived risks to the privacy of trial participants who might not have consented to their data being shared, and the availability of resources for archiving the data. As a result, in the latest update to its guidance, in 2017, it settled for merely encouraging data sharing and requiring statements about whether and where data would be shared.

oldgradstudent 3y ago

With the risk of starting a flame war, we recently had several well-publicised clinical trials that reported 95% efficacy of some certain modality. Yet, in reality, efficacy as defined in the trial turned out to be closer to 0%. Instead of investigating what in the design and execution of the trial led to such a discrepancy, the problem was handled by denying there was a problem, changing the goalposts, reporting ad-hoc hypotheses as facts, silencing all critics, and forcing the public to take the modality anyway or lose jobs, school, and freedom of movement.

taeric 3y ago
I can only guess you mean something regarding the pandemic? Do you have links to show things were "closer to 0%?" Sounds more than a touch outlandish. :( I'm confident we will know even more about things as time goes on. I'm less confident on any nefarious motivations in most of it. Reality is that a lot of people died, and everyone was trying to gain control and an advantage over the situation. Mistakes were certainly made, but I am back to low confidence in thinking that everything was a mistake.
- oldgradstudent 3y ago
  >Do you have links to show things were "closer to 0%?" Sounds more than a touch outlandish. :( Take the Pfizer vaccine. The clinical trial's main endpoint was ~95% efficacy in one thing and one thing only, prevention of symptomatic Covid. Not reduction in mortality, not reduction in serious disease, not infection, and not spread. The only thing the trial tested and reported was prevention of symptomatic Covid. This is also the sole indication in the package insert as approved by the FDA. In reality, everyone I know got vaccinated and got symptomatic Covd. I mean everyone, no exceptions. The situation is the similar in my entire country and around the world.
  - taeric 3y ago
    Do you know of any studies on the discrepancy? My understanding was that Omicron came out and basically gave the middle finger to everyone's precautions. With what seemed like literally nothing working against it.

mydriasis 3y ago

More and more it seems that science is being clouded by moneyed interests and greed. If we can't trust science, what can we trust?

thechao 3y ago
This is just reporting bias. I've worked (on-and-off) in various scientific fields for ~30; it has always had its bad actors. (I even helped do the statistics for some!) I'd urge you to consider following the situations: 1. Prescientific inquiry; and, 2. PreFDA food, drug, and medicine. Both of those were orders-of-magnitude worse than what we have, now. Could we do better? Sure! Is it broken? No.
- mydriasis 3y ago
  That brings a bit of hope!
- LorenPechtel 3y ago
  Disagree. Broken isn't a binary--the current system is far better than what came before, but that doesn't mean there aren't serious flaws in the current system.
jasmer 3y ago
[dead]

gigatexal 3y ago

In the age of QAnon and Alex Jones and science and vaccine deniers we can’t have things like clinical studies be corrupted. Ugh.

sonicshadow 3y ago
Yeah not a great look, maybe science isn’t the truth after all
linuxftw 3y ago
The 'vaccine deniers' have been calling the 'studies' and 'trials' trash for a long time. The evidence comes out that they're trash, and somehow the 'vaccine deniers' are still wrong? They removed liability from manufacturers, and suddenly the CDC schedule exploded with new products. I mean, chicken pox has a vaccine now?
- nocoolnametom 3y ago
  That which was asserted without evidence could correctly be dismissed without evidence. AT THE TIME the anti-vax crowd was basing their positions entirely upon anecdote, rumor, and often badly misread prepublication research and stats. Their methodology was inherently flawed. Even if the conclusions they came to have been "validated" their position was still built upon this same flimsy scaffolding. It's not like the "do your own research" blogs and videos somehow gathered the same evidence used by this paper. This also does not indicate that other positions held by the same crowd, which are similarly based upon "anecdata" and rumor, are somehow made more evident by this paper in Nature.
  - linuxftw 3y ago
    I agree, the assertion that 'vaccines' are safe can be dismissed without evidence. There's no evidence concluding they're actually safe. In fact, we have given the manufacturers immunity because they're 'unavoidably unsafe.' Just look at how the COVID trials were conducted. They didn't even test each patient. Only some patients that presented symptoms, and then not even all of those patients. How long did they follow the health outcomes for approved vaccines in the test groups? 3 months at most, and many trials, not even that long. So if someone suffers a neurological condition, well, we just won't know about it.

dablweb 3y ago

Ironic that nature publishes this despite being guilty of constantly promoting it.

jmpeax 3y ago

Raw data examined: Ok 56%, Flawed 18%, Zombie 26% Raw data not available: Ok 97%, Flawed 2%, Zombie 1% Perhaps it's not good to call an unknown as "Ok"? Maybe Carlisle should add his own paper to the mix?

twic 3y ago
That's literally the point he's making! The terms "flawed" and "zombie" reflect positive identification of dodgy data, so of course when the data isn't available, they are less likely to apply, hence why: > This finding alarmed him, too: it suggested that, without access to the IPD — which journal editors usually don’t request and reviewers don’t see — even an experienced sleuth cannot spot hidden flaws.

deleted 3y ago

[deleted]

gumby 3y ago

Note that the article discusses research studies and not clinical trials for drug or device approval. These research studies are important (look at how many were conducted on COVID-19 over the last few years) but are typically not held to a particularly high standard, as with most science. Which doesn’t excuse bad data or poor statistics (the latter supposedly supposed to be picked up in peer review).

piqufoh 3y ago
Hmm, I read the article as explicitly calling out "clinical trials" (as referenced in the title and abstract) and it makes no reference research studies. I don't understand the distinction between "research studies" and "clinical trials", surely all research studies where an RCT is performed with real patients and real drugs is a clinical trial?
- gumby 3y ago
  I meant “trials for research studies” as opposed to “trials for drug or device approval.” The amount of record keeping and oversight of a drug approval trial is enormous (and as a consequence insanely expensive) — data handling, having disjoint groups at each stage handling and analyzing data, etc and detailed records of every manufacturing step — think ISO9000 on steroids. Nobody would bother to go to that effort for a scientific exploration, nor should they. So the bar is much lower. I am making no excuse for shoddy science! But it is quite unlikely for a licensed drug.

freedude 3y ago

“I think journals should assume that all submitted papers are potentially flawed and editors should review individual patient data before publishing randomised controlled trials,” Carlisle wrote in his report. This should be considered part of every journal's idea of due diligence and this shouldn't be a new idea. Shysters, con artists and snake oil salesmen have been around for a long time. The purpose of a Journal is to publish reliable information and weed out the garbage. How can you do that if you are not looking at the entire picture?

isaacremuant 3y ago

No. no. This is not permitted. Trust the science or you're an antivaxer, Trumper, denier, racist, white supremacist. Media and gov, or media at the behest of gov (as the twitter files prove, but it was obvious without them), censored as much as thet could, everything that went against what they wanted to push. It wasn't science but security/hygiene theater and it worked, because people did go along and did turn on their neighbours who opposed the measures. But now we slowly get tidbits of things we can debate again ... Funny that, but no recognition that the entire lockdown, masks and vaccine mandate effort + economic destruction and theft (tax money to corps as "aid") was never a reasonable, logical or scientific response. It was an authoritarian and corrupt response of extreme Effectiveness and cynicism. But here comes some guy to say "people have always quarantined in pandemics" or some other disingenuous claim that ignores the reality of what happened: healthy people denied basic human and constitutional rights.

LorenPechtel 3y ago
The problem here is "healthy people". Covid's real key to success is the fact that most spread is presymptomatic. It's the apparently healthy people spreading it! And note the lessons of history: 1) There will always be those who choose short term economic interests over safety when the threat isn't absolutely proven. They'll always close the barn door too late. 2) Places that take epidemic/pandemic threats seriously tend to fare better economically in the long run.
- isaacremuant 3y ago
  No. The problem is that fundamentally, you and people like you decided that authortiarianism is ok if the gov says "things are scary" and everyone should suffer the consequences. No debate allowed. The risk profile for people was always ridículous and you were never going to contain it once it was widespread, which it was, but somehow we believed in the rolling "it's just 2 weeks". 1) it wasn't short term. It wasn't safety. Short and long term you hurt the vulnerable and you helped the rich and powerful. 2) Rich places will keep being richer and poor places poorer, and when the powers that be decided that enough was enough, all the concerns of the hypochondriacs suddenly didn't matter. You could tell, if you paid attention, that politicians weren't afraid after the initial surge, but wanted "the masses" to be. You even have definite proof in many places, one of them being UK and number 10. Can't link it now, sorry. But Google downing Street covid rules or something on that note and you can probably find a lot. It wasn't limited to the UK. It was everywhere you looked properly. You were scammed and you either were well off and didn't mind that much or you want to pretend you weren't for your own mental health. Because the truth is a hard pill to swallow.

xhkkffbf 3y ago

Even if they aren't faked, the ability to shut down a trial that isn't delivering the right preliminary data is a big problem for society. Why should we trust these drugs?

throw9away6 3y ago
A lot of these studies are prerun in the 3rd world before the real one is done for credit. If sideffects show up the drug can sometimes be mixed with one that has the same known side effects to fool the studies.

Roark66 3y ago

They say "medicine". I would say science in general, perhaps we could generalise even further to "any human activity is full of unethical people trying to exploit it". But 25%!? That suggest there is a big problem with how we "do science". Unfortunately I have no solution to the problem. Publishing everything (including raw data) for every research would probably help somewhat, but only teams repeating experiments/trials would ensure it. I wonder if we suddenly took 10% of all money spent on science (let's say in medicine) and instead of novel research we used it to redo randomly chosen previous research. Would we loose or gain in terms of new cures? And if we gained, what if we spent 15%, or 25%? That's a great idea for a scientific study to find a point of diminishing returns on "research verification". Would someone please write a research grant request for this?

davidktr 3y ago
Of course there are big problems with how we do science. Much of it is garbage. Imagine most software was written by junior engineers, without any code review or input from seniors. That is today's science. Most scientific legwork is done by absolute beginners, i.e., graduate students. They often lack a support structure to focus on what they have learned so far. Most of the world is not Oxbridge, MIT, Stanford. Where are the beginners' supervisors, you might ask? Chasing the latest trend to secure funding. Pondering how their line of research can be formulated as buzzword-du-jour markov chain. Ass kissing the dean to get department funding. Having worked in research for 15 years, I am certain about two things: (1) The scientific method yields better results than doing things freehand. (2) Randomly axing 50% of academia would improve the situation.
bobbylarrybobby 3y ago
That 10% investment would have a huge payoff too because it would shut down avenues that were only opened due to p-hacking long before they'd had a chance to seek further investment.

deleted 3y ago

[deleted]

j-pb 3y ago

That and lawyer driven development. I have multiple sclerosis, and I never know if a new medication came out because it's actually better or simply because they could get a new patent for a slightly modified molecule.

LorenPechtel 3y ago
Disagree--you're talking about patent-driven development, not lawyer-driven development.

thenerdhead 3y ago

Another good reason why you should learn how to read studies: https://peterattiamd.com/ns001/ https://peterattiamd.com/ns001/ https://biolayne.com/reps/how-to-read-research-a-biolayne-guide/ https://biolayne.com/reps/how-to-read-research-a-biolayne-gu... https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7392212/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7392212/ And as a bonus, read books by controversial figures who talk about these challenges through their own published studies and decided it isn't worth fighting. What would be cool is to see "trust indicators" as part of a study's metadata as it is reviewed through time and continually shared by others. Could be a "study health score" or a checklist showing the study isn't biased heavily by sponsors, methods, or misleading language. Especially given that medicine is becoming more personal with the advancements of AI and accessibility of tests, someone should be able to understand the health of a study at-a-glance reviewed by peer reviews rather than disclosed only by the authors.

randcraw 3y ago
A primary trust indicator of any research result is the reputation of the journal publishing the paper. Better journals demand more, not just in terms of the impact/insight of the results, but also the rigor of the experimental methodology used and how well the data was curated and the confounding variables identified and isolated. And of course, the criteria for observational-based research differ a lot from mechanism-based, especially since the former can't control for nearly as many variables. The same goes for simulations or interpretive modeling, where experimentation is minimal. Research studies differ quite a lot in how precisely the mechanism of action is identified, isolated, tested, and results interpreted. IMHO, blackening the trustworthiness of all of science is unhelpful, especially when some models and methods are surely more trustworthy and replicable than others. Better to identify and catalog specific sources of error (or imprecision) in order to remedy them than to just toss the baby.

darkclouds 3y ago

I'd trust the NHS if the National Institute of Clinical Excellence (NICE) actually published their minutes online. Most employees only know what they have been taught, so whilst I agree with the headline that some trial data is dodgy, it doesnt just end there. There are multiple pathways to factor in, there is redundancy built in, ie secondary pathways, there is the fact that not all chemicals go where intended (best highlighted by radioactive isotopes). When looking at the history of patented medicine, this really started between WW1 and WW2, before WW1, most GP's prescribed what was found in the body on a like for like basis, and in some cases prescribed organs in various forms, like desiccated thyroid gland for thyroid related problems, for pernicious anaemia, raw liver used to be prescribed to women and so on, but that has its problems like contamination and diseases. So from WW1/WW2 onwards the rise of patented medicine took hold, but the main problem with patented medicine is the human body hasnt evolved to use these new chemical compounds in the same way as unpatented chemicals which have been around for thousands of years. And todays GP's dont really highlight the side effects of the patented medicine, and because they simply dont ask what you have been eating and drinking etc, they have this hubris which sucker punches your trust and sucks you in like a black hole, until before you know it you are on half a dozen different drugs, your quality of life is going down the pan and you've been left a zombie wondering where did Hitler go wrong modelling the German state of the time on the British Empire? You Americans complaining about the cost of healthcare, should count yourself lucky those insurance companies are looking out for your long term interests and theirs! Fortunately, hospitals in other countries publishes studies and as english seems to be the main language used for science, I have to tip my hat to the Chinese who are roaring up the charts in terms of investigating and publishing relevant studies that will complement a quality of life one hopes to achieve, and we cant forget Wikipedia, Pubmed and Google for connecting users with pertinent studies. Saying that I do sometimes wonder if something like ChatGPT has written some studies due to the poor quality of english used, but generally they stand out like a sore thumb. Anyway, does any know why there is a connection between MDMA, blood clotting and Manganese?

amai 3y ago

There should be a paperswithdata for medicine like there is a https://paperswithcode.com/ https://paperswithcode.com/ for data science.

jet_32951 3y ago

Have a look at Derek Lowe's excellent blog [0] to view the depths of malfeasance, if not outright fraud, in which "studies" are created. [0] https://www.science.org/content/blog-post/fakin-it-modern-way https://www.science.org/content/blog-post/fakin-it-modern-wa... is just his latest in a long string of well-documented posts.

OnlyMortal 3y ago

To be fair, science is now a “publish or be damned” business. So many papers are not worthy of publication and, frankly, are of common knowledge anyway. Of course, this is driven by the money backing the research which, I’m sure, this is why clinical trials in the medical areas can be of a poor quality. People want to keep their jobs.

tails4e 3y ago

Nutrition for me it a big issue. It seems like and ideal area for scientific and medical study to give us light, but seems so hard to get truly objective info. Fat is the devial/good/bad/OK, sugar is the devil/bad/ok, etc. Should I avoid white bread like a hole in the head, or is it fine? Too many agendas and not enough truth.

jononomo 3y ago

I don't think it makes sense to publish the results of clinical trials and other scientific experiments until after they have been independently replicated.

Madmallard 3y ago

This is entropy in action. We're not going to move forward as long as malicious behavior is rewarded with money in large-scale systems. I suspect in coming decades more incurable chronic illness will become rampant and polypharmacy will become more common and necessary for larger percentages of the population. It is really sad to see.

dennis_jeeves1 3y ago

I will start my own medical 'system' and be my own doctor. To hell with the mainstream and it's corrupt/stupid adherents.

pyuser583 3y ago

It doesn’t help that front-line doctors are expected to publish.

NoZebra120vClip 3y ago

It was interesting to find out that test subjects who pass away during clinical trials are excluded, erased, from the trial records. Their participation and data is not included in the final results, as if they were never in the trial to begin with. Many pharmaceutical companies have policies which prohibit patients from participating in studies for anti-depressants if they have suicidal ideations. In practice, this significantly reduces the participation of any patient with depression.