12 ms·

"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these tr

"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either because the authors were incompetent, or because they had faked the data." So, 70% fake/flawed. The finding falls in line with other large scale replication studies in medicine, which have had replication success rates ranging from 11 to 44%. [1] It's quite difficult to imagine why studies where a positive outcome is a gateway to billions of dollars in profits, while a negative outcome would result in substantial losses, might end up being somehow less than accurate. [1] - https://en.wikipedia.org/wiki/Replication_crisis#In_medicine https://en.wikipedia.org/wiki/Replication_crisis#In_medicine

cypress66 3y ago
You don't need "big pharma" for this. Researchers at universities also do these kinds of things because it helps them advance their careers.
- jermaustin1 3y ago
  >helps them advance their careers ... into the multi-billion dollar companies GP is talking about.
  - Tenoke 3y ago
    A lot who stay purely in Academia do it as well, presumably for the prestige.
    - Ilverin 3y ago
      Well you need peer reviewed papers in highish impact journals to get tenure. For some people, the only way they're getting that is by cheating.
      - BurningFrog 3y ago
        Doing research has a lottery element. You explore something that might reveal an important discovery. And sometimes it just doesn't. That doesn't mean you're a bad scientist, just an unlucky one. But it does mean you can't get tenure. So it's easy to understand why people fake results to secure a career.
        autoexec 3y ago
        > That doesn't mean you're a bad scientist, just an unlucky one. But it does mean you can't get tenure. That sounds like a really easy problem to solve. Just treat valid science as important regardless of the results. The results shouldn't matter unless they've been replicated and verified anyway.
        vitalurk 3y ago
        Too easy. Define "valid" though.
        autoexec 3y ago
        Valid as in meaningfully peer reviewed to avoid flawed/badly designed studies as well as total garbage (for example https://nerdist.com/article/fake-star-wars-midi-chlorian-paper-fools-4-science-journals/ https://nerdist.com/article/fake-star-wars-midi-chlorian-pap...) but the gold standard should be replication. We should reward quality work, not simply the number of research papers (since it's easy to churn out trash) or what the results are (because until they are verified they could be faked).
        vitalurk 3y ago
        I wholeheartedly agree.
        BurningFrog 3y ago
        It's a really easy solution to describe. Actually implementing it across the academic world seems much harder.
    - rqtwteye 3y ago
      I would think a successful study is way better for your academic career than a failed study.
      - ethanbond 3y ago
        Which is why you fake your data into looking successful, and you don’t even go through the effort of publishing the failed studies (leading to its own additional problems in understanding what’s true and what we know)
- LMYahooTFY 3y ago
  What's your point? We have "big pharma" for this. Do you think CocaCola and the Sacklers had their own unique ideas shared by no one else? That we've filtered all scrupulous people out of industry? Scruples are an abstraction at that scale.
  - smcin 3y ago
    Is the Coca-Cola mention about their campaign to influence CDC research and policy [0], and pro-obesity astroturf [1][2]? [0]: Politico "Coca-Cola tried to influence CDC on research and policy, new report states" [https://www.politico.com/story/2019/01/29/coke-obesity-sugar-research-1125003 https://www.politico.com/story/2019/01/29/coke-obesity-sugar...] [1]: "Evaluating Coca-Cola’s attempts to influence public health ‘in their own words’: analysis of Coca-Cola emails with public health academics leading the Global Energy Balance Network" https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10200649/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10200649/ [2]: Forbes: "Emails Reveal How Coca-Cola Shaped The Anti-Obesity Global Energy Balance Network" https://www.forbes.com/sites/nancyhuehnergarth/2015/11/24/emails-reveal-how-coca-cola-shaped-the-anti-obesity-global-energy-balance-network https://www.forbes.com/sites/nancyhuehnergarth/2015/11/24/em...
- smu3l 3y ago
  > Researchers at universities also do these kinds of things because it helps them advance their careers. This is a huge problem and in my opinion is mostly due to bad incentive structures and bad statistical/methodological education. I'm sure there are plenty of cases where there is intentional or at least known malpractice, but I would argue that most bad research is done in good faith. When I was working on a PhD in biostatistics with a focus on causal inference among other things, I frequently helped out friends in other departments with data analysis. More often than not, people were working with sample sizes that are too small to provide enough power to answer their questions, or questions that simply could not be answered by their study design. (e.g. answering causal questions from observational data*). In once instance, a friend in an environmental science program had data from an experiment she conducted where she failed to find evidence to support her primary hypothesis. It's nearly impossible to publish null results, and she didn't have funding to collect more data and had to get a paper out of it. She wound up doing textbook p-hacking; testing a ton of post-hoc hypotheses on subsets of data. I tried to reel things back but I couldn't convince her to not continue because "that's how they do things" in her field. In reality she didn't really have a choice if she wanted to make progress towards her degree. She was a very smart person, and p-hacking is conceptually not hard to understand, but she was incentivized to not understand it or to not look at her research in that way. * Research in causal inference is mostly about rigorously defining the (untestable) causal assumptions you must make and developing methods to answer causal questions from observational data. Even if an argument can be made that you can make those assumptions in a particular case, there is another layer of modeling assumptions you'll end up making depending on the method you're using. In my experience it's pretty rare that you can really have much confidence that your conclusions about a causal question if you can't run a real experiment.
  - bigfudge 3y ago
    It’s so interesting to hear you say that. I became disillusioned with causal methods for observational data for similar reasons. You can’t often model your way to interesting inferences without an experiment.
Zalastax 3y ago
Is there not full or at least partial overlap between the 44 % and the 26 %? Which would mean not 70 % but some smaller number?
morelisp 3y ago
> The finding falls in line with other large scale replication studies in medicine, which have had replication success rates ranging from 11 to 44%. These numbers seem almost wholly unrelated. A perfectly good study may be extremely difficult to replicate (or even the original purpose of replication - the experiment as described in the paper may simply not be sufficient); and an attempt at replication (or refutation), successful or not, is under the same pressure to be faked or flawed as the original paper.
qt31415926 3y ago
I don't think the statement reads that the 44% and the 26% should be additive. Especially given the zombie graphic where it looks like they overlap the 26 on top of the 44, where the orange bar is the 26% and the remaining yellow bar is the 44%
NotHowStatsWork 3y ago
"So, 70% fake/flawed." I think you miss read that section? Only were 44% fake or flawed acording to this study. The 26% that were very flawed is a subset of the 44% that were flawed in general. So those precentages should not be added together.
- cycomanic 3y ago
  Oh the irony!
- queuebert 3y ago
  Are you saying the comment's analysis was ... flawed?
adasdasdas 3y ago
I also see similar findings in tech where most experiment results are "fudged". In some cases, people run the "same" experiment 5+ times until one is stat sig.
paulusthe 3y ago
It's more complex than just that. Sure, there's the people trying to make a dollar who are willing to do bad science in order to get the result they want. But there's also the general publication bias against replication studies - who wants to read them, and who wants to do them (they're not usually seen as prestigious academically: most academics want to test their ideas, not those of others. And then there's cultural differences in which people sometimes see a negative result as a "failure", don't publish it as a result, and instead skew the data and lie their asses off in order to gain prestige in their career. As long as nobody double checks you, you're good.
- autoexec 3y ago
  > ut there's also the general publication bias against replication studies - who wants to read them, and who wants to do them (they're not usually seen as prestigious academically: most academics want to test their ideas, not those of others. Academia seems like the idea place for this. Why not require a certain number of replicated studies in order to get a degree? Universities could then be constantly churning out replication studies. More importantly, why do we bother taking anything that hasn't been replicated seriously? Anyone who publishes a paper that hasn't been verified shouldn't get any kind of meaningful recognition or "credit" for their discovery until it's been independently confirmed. Since anyone can publish trash, having your work validated should be the only means of gaining prestige in your career.
  - cm2012 3y ago
    Maybe a school should start giving PhDs for good replications.
costigan 3y ago
I think you misread the article slightly. The 26% of zombie papers are inside the 44% of papers that contained at least some flawed data. Look at the figure just below this quote where the blue bar, indicating papers he thought were ok, covers more than 50%.
- crazygringo 3y ago
  Which, ironically, just shows how easy it is to make mistakes not just in complicated statistical methods, but in basic interpretation of what numbers mean in the first place.
  - SubiculumCode 3y ago
    not to mention that everyone here forgets to mention that the majority of submitted papers with suspected fraud came from countries with known issues with fraudulent science publishing. e.g. China. The USA, for example, looked much better (although not perfect) at 6% zombie.
    - robwwilliams 3y ago
      This. Scientific corruption probably correlates well with political corruption by nation and region.
onlyrealcuzzo 3y ago
As someone who works in Data Science at FAANG - if you look hard enough - there is something questionably wrong in every step of the data funnel. And that's when I believe people do have a somewhat best effort to maximize profits. There are plenty of people that only care about career progression and think they can get away with lying and cheating their way to the top. They wouldn't believe that if it didn't work sometimes. These medical studies are also run mainly to maximize profits, also by some career climbers. They are not run virtuously for the betterment of society. So I would be astounded if they are as reliable as people might like to believe. Maybe I'm just being grossly skeptical. Actually, I'd feel better if someone could convince me I'm completely unfounded here.
- shaburn 3y ago
  Really curious how a profitable industry like medical research ranks against less profitible.
- verisimi 3y ago
  I don't think that's skeptical. I do think there is an even worse issue - which is funding. The money incentive means you can fund studies that support whatever you want.
- ancorevard 3y ago
  "And that's when I believe people do have a somewhat best effort to maximize profits." Nope. I actually think that if you do scientific research as a company (profit) it may make you less bad/less likely to do fraud compared to academia (non-profit). Reason is that there are more ways to punish you, employees, board, investors, etc in a profit seeking vehicle, and as a profit seeking vehicle being caught must be part of the profit seeking calculation – in the end, the world of reality/physics will weigh your contribution. I believe there is evidence that there is more fraudulent scientific research happening in non-profit vehicles/academia. Take for example an area where there are fewer profit seeking companies participating - social sciences. It's dominated by academia. Now look at the replication rate of social sciences.
  - 3y ago
codingdave 3y ago
> Carlisle got access to anonymized individual participant data (IPD) I'm not in the industry so my question might have an obvious answer to those of you who are: How would one go about getting IPD if you wanted to run your own analysis of trial data or other data-driven research?
- misterdad 3y ago
  You'll need to reach out to the study authors with a request. If they are interested (you're going to publish something noteworthy with a citation for them (low chance), you want to bring them in on some funded research (better chance), etc) then they'll push it to their Institutional Review Board (a group of usually faculty and sometimes administrative staff at a University/Hospital/Org) who will review the request, the conditions of the initial data collection, legal restrictions, and then decide if they'll proceed with setting up some sort of IRB agreement / data use agreement. Unless you're a tenured professor somewhere or a respected researcher with some outside group then you probably won't get past any of those steps. Even allegedly anonymized data comes with the risk of exposure (and real penalties) not to mention the administrative overhead (expense, time, attention) that you'll need to be able to cover the cost of through some funded research. That research, btw, will also need to be through some IRB structure. You can tap a private firm that acts as an IRB but that's another process entirely and most certainly requires fat stacks of cash. Legal privacy concerns, ethical concerns, careerism (nobody wants you to find the 'carry the two' you forgot so you can crash their career prospects), bloated expenses (somebody has to pay for all of that paperwork, all those IRB salaries, etc) and etc, etc, etc all keep reproducibility of individual data frozen. Even within the same institution. Within the same team! You have to tread lightly with reproduction.
deleted 3y ago
[deleted]
SubiculumCode 3y ago
I'd like to point out that the study [1] reported 16 trials from the USA with just one "zombie/fatally flawed" trial. e.g. 6% Most of the studies that were problematic came from China and Egypt. In other words, nothing new here. [1] https://associationofanaesthetists-publications.onlinelibrary.wiley.com/doi/10.1111/anae.15263 https://associationofanaesthetists-publications.onlinelibrar...
waterheater 3y ago
Two relevant perspectives to share: "It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of the New England Journal of Medicine." -Marcia Angell "If the image of medicine I have conveyed is one wherein medicine lurches along, riven by internal professional power struggles, impelled this way and that by arbitrary economic and sociopolitical forces, and sustained by bodies of myth and rhetoric that are elaborated in response to major threats to its survival, then that is the image supported by this study." -Evelleen Richards, "Vitamin C and Cancer: Medicine or Politics?"
- avalys 3y ago
  Ah, but don’t worry folks, those vaccines which generated $40B in profits for Pfizer and Moderna, those are just A-ok. Trust the science, which in this case is perfect, unassailable and uncorruptible (unlike everything else the pharma industry does). Don’t question the faith.
  - chiefalchemist 3y ago
    And those weren't simple studies they were self-policed and self-reported trials. Trials that determined FDA green lighting. How can the FDA operate like that when the incentives to profit are 40B x more than being honest. File under: Follow the money
  - strken 3y ago
    "Trust the science" is dumb, but "trust that the single-most reviewed medical studies of the past decade with 30k+ participants aren't outright bullshit" is enough for me. The risk for those studies is probably closer to what happened with Astrazeneca, where a very rare side effect got missed because 30k people weren't enough to detect it. I think skepticism is healthy to a degree, but ironically, the more adversarial the general public is to an area of study the more rigorous it usually is.
    - 3y ago
dimal 3y ago
I read those two percentages as not being added to each other. In other words, the 26% is a shittier subset of the 44%. At least that's what I think they mean.
toss1 3y ago
What I wonder is, since it is so easy for this one researcher to catch these flaws and lies, why are these not being caught by peer review? Increasing seems like peer review is a waste of everyone's time, and it's be better to just publishing when you think your ready on the likes of arXiv, and let everyone examine and criticize it. Run it up the flagpole, see if it gets saluted or shot down. Seems like there's be much more of an incentive to run confirmation studies.
ekianjo 3y ago
> It's quite difficult to imagine why studies Why? Because its just cargo cult science at the end of the day. Once you incentivize clinical trials to be used as a tool to unlock massive profits through an arbitrary process with a single regulator as judge it WILL be hacked. At every possible part of the process including regulators.
enkid 3y ago
Does the 44% not include the 26%?
eru 3y ago
> So, 70% fake/flawed. It's worse. First, there's a selection bias in what trials the author can get data for. Second, the 44% + 26% are just those where the author can detect problems from reading their data alone. If someone convincingly fakes their data, the author can't detect that.

cypress66 3y ago

You don't need "big pharma" for this. Researchers at universities also do these kinds of things because it helps them advance their careers.

jermaustin1 3y ago
>helps them advance their careers ... into the multi-billion dollar companies GP is talking about.
- Tenoke 3y ago
  A lot who stay purely in Academia do it as well, presumably for the prestige.
  - Ilverin 3y ago
    Well you need peer reviewed papers in highish impact journals to get tenure. For some people, the only way they're getting that is by cheating.
    - BurningFrog 3y ago
      Doing research has a lottery element. You explore something that might reveal an important discovery. And sometimes it just doesn't. That doesn't mean you're a bad scientist, just an unlucky one. But it does mean you can't get tenure. So it's easy to understand why people fake results to secure a career.
      - autoexec 3y ago
        > That doesn't mean you're a bad scientist, just an unlucky one. But it does mean you can't get tenure. That sounds like a really easy problem to solve. Just treat valid science as important regardless of the results. The results shouldn't matter unless they've been replicated and verified anyway.
        vitalurk 3y ago
        Too easy. Define "valid" though.
        autoexec 3y ago
        Valid as in meaningfully peer reviewed to avoid flawed/badly designed studies as well as total garbage (for example https://nerdist.com/article/fake-star-wars-midi-chlorian-paper-fools-4-science-journals/ https://nerdist.com/article/fake-star-wars-midi-chlorian-pap...) but the gold standard should be replication. We should reward quality work, not simply the number of research papers (since it's easy to churn out trash) or what the results are (because until they are verified they could be faked).
        vitalurk 3y ago
        I wholeheartedly agree.
        BurningFrog 3y ago
        It's a really easy solution to describe. Actually implementing it across the academic world seems much harder.
  - rqtwteye 3y ago
    I would think a successful study is way better for your academic career than a failed study.
    - ethanbond 3y ago
      Which is why you fake your data into looking successful, and you don’t even go through the effort of publishing the failed studies (leading to its own additional problems in understanding what’s true and what we know)
LMYahooTFY 3y ago
What's your point? We have "big pharma" for this. Do you think CocaCola and the Sacklers had their own unique ideas shared by no one else? That we've filtered all scrupulous people out of industry? Scruples are an abstraction at that scale.
- smcin 3y ago
  Is the Coca-Cola mention about their campaign to influence CDC research and policy [0], and pro-obesity astroturf [1][2]? [0]: Politico "Coca-Cola tried to influence CDC on research and policy, new report states" [https://www.politico.com/story/2019/01/29/coke-obesity-sugar-research-1125003 https://www.politico.com/story/2019/01/29/coke-obesity-sugar...] [1]: "Evaluating Coca-Cola’s attempts to influence public health ‘in their own words’: analysis of Coca-Cola emails with public health academics leading the Global Energy Balance Network" https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10200649/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10200649/ [2]: Forbes: "Emails Reveal How Coca-Cola Shaped The Anti-Obesity Global Energy Balance Network" https://www.forbes.com/sites/nancyhuehnergarth/2015/11/24/emails-reveal-how-coca-cola-shaped-the-anti-obesity-global-energy-balance-network https://www.forbes.com/sites/nancyhuehnergarth/2015/11/24/em...
smu3l 3y ago
> Researchers at universities also do these kinds of things because it helps them advance their careers. This is a huge problem and in my opinion is mostly due to bad incentive structures and bad statistical/methodological education. I'm sure there are plenty of cases where there is intentional or at least known malpractice, but I would argue that most bad research is done in good faith. When I was working on a PhD in biostatistics with a focus on causal inference among other things, I frequently helped out friends in other departments with data analysis. More often than not, people were working with sample sizes that are too small to provide enough power to answer their questions, or questions that simply could not be answered by their study design. (e.g. answering causal questions from observational data*). In once instance, a friend in an environmental science program had data from an experiment she conducted where she failed to find evidence to support her primary hypothesis. It's nearly impossible to publish null results, and she didn't have funding to collect more data and had to get a paper out of it. She wound up doing textbook p-hacking; testing a ton of post-hoc hypotheses on subsets of data. I tried to reel things back but I couldn't convince her to not continue because "that's how they do things" in her field. In reality she didn't really have a choice if she wanted to make progress towards her degree. She was a very smart person, and p-hacking is conceptually not hard to understand, but she was incentivized to not understand it or to not look at her research in that way. * Research in causal inference is mostly about rigorously defining the (untestable) causal assumptions you must make and developing methods to answer causal questions from observational data. Even if an argument can be made that you can make those assumptions in a particular case, there is another layer of modeling assumptions you'll end up making depending on the method you're using. In my experience it's pretty rare that you can really have much confidence that your conclusions about a causal question if you can't run a real experiment.
- bigfudge 3y ago
  It’s so interesting to hear you say that. I became disillusioned with causal methods for observational data for similar reasons. You can’t often model your way to interesting inferences without an experiment.

Zalastax 3y ago

Is there not full or at least partial overlap between the 44 % and the 26 %? Which would mean not 70 % but some smaller number?

morelisp 3y ago

> The finding falls in line with other large scale replication studies in medicine, which have had replication success rates ranging from 11 to 44%. These numbers seem almost wholly unrelated. A perfectly good study may be extremely difficult to replicate (or even the original purpose of replication - the experiment as described in the paper may simply not be sufficient); and an attempt at replication (or refutation), successful or not, is under the same pressure to be faked or flawed as the original paper.

qt31415926 3y ago

I don't think the statement reads that the 44% and the 26% should be additive. Especially given the zombie graphic where it looks like they overlap the 26 on top of the 44, where the orange bar is the 26% and the remaining yellow bar is the 44%

NotHowStatsWork 3y ago

"So, 70% fake/flawed." I think you miss read that section? Only were 44% fake or flawed acording to this study. The 26% that were very flawed is a subset of the 44% that were flawed in general. So those precentages should not be added together.

cycomanic 3y ago
Oh the irony!
queuebert 3y ago
Are you saying the comment's analysis was ... flawed?

adasdasdas 3y ago

I also see similar findings in tech where most experiment results are "fudged". In some cases, people run the "same" experiment 5+ times until one is stat sig.

paulusthe 3y ago

It's more complex than just that. Sure, there's the people trying to make a dollar who are willing to do bad science in order to get the result they want. But there's also the general publication bias against replication studies - who wants to read them, and who wants to do them (they're not usually seen as prestigious academically: most academics want to test their ideas, not those of others. And then there's cultural differences in which people sometimes see a negative result as a "failure", don't publish it as a result, and instead skew the data and lie their asses off in order to gain prestige in their career. As long as nobody double checks you, you're good.

autoexec 3y ago
> ut there's also the general publication bias against replication studies - who wants to read them, and who wants to do them (they're not usually seen as prestigious academically: most academics want to test their ideas, not those of others. Academia seems like the idea place for this. Why not require a certain number of replicated studies in order to get a degree? Universities could then be constantly churning out replication studies. More importantly, why do we bother taking anything that hasn't been replicated seriously? Anyone who publishes a paper that hasn't been verified shouldn't get any kind of meaningful recognition or "credit" for their discovery until it's been independently confirmed. Since anyone can publish trash, having your work validated should be the only means of gaining prestige in your career.
- cm2012 3y ago
  Maybe a school should start giving PhDs for good replications.

costigan 3y ago

I think you misread the article slightly. The 26% of zombie papers are inside the 44% of papers that contained at least some flawed data. Look at the figure just below this quote where the blue bar, indicating papers he thought were ok, covers more than 50%.

crazygringo 3y ago
Which, ironically, just shows how easy it is to make mistakes not just in complicated statistical methods, but in basic interpretation of what numbers mean in the first place.
- SubiculumCode 3y ago
  not to mention that everyone here forgets to mention that the majority of submitted papers with suspected fraud came from countries with known issues with fraudulent science publishing. e.g. China. The USA, for example, looked much better (although not perfect) at 6% zombie.
  - robwwilliams 3y ago
    This. Scientific corruption probably correlates well with political corruption by nation and region.

onlyrealcuzzo 3y ago

As someone who works in Data Science at FAANG - if you look hard enough - there is something questionably wrong in every step of the data funnel. And that's when I believe people do have a somewhat best effort to maximize profits. There are plenty of people that only care about career progression and think they can get away with lying and cheating their way to the top. They wouldn't believe that if it didn't work sometimes. These medical studies are also run mainly to maximize profits, also by some career climbers. They are not run virtuously for the betterment of society. So I would be astounded if they are as reliable as people might like to believe. Maybe I'm just being grossly skeptical. Actually, I'd feel better if someone could convince me I'm completely unfounded here.

shaburn 3y ago
Really curious how a profitable industry like medical research ranks against less profitible.
verisimi 3y ago
I don't think that's skeptical. I do think there is an even worse issue - which is funding. The money incentive means you can fund studies that support whatever you want.
ancorevard 3y ago
"And that's when I believe people do have a somewhat best effort to maximize profits." Nope. I actually think that if you do scientific research as a company (profit) it may make you less bad/less likely to do fraud compared to academia (non-profit). Reason is that there are more ways to punish you, employees, board, investors, etc in a profit seeking vehicle, and as a profit seeking vehicle being caught must be part of the profit seeking calculation – in the end, the world of reality/physics will weigh your contribution. I believe there is evidence that there is more fraudulent scientific research happening in non-profit vehicles/academia. Take for example an area where there are fewer profit seeking companies participating - social sciences. It's dominated by academia. Now look at the replication rate of social sciences.
- 3y ago

codingdave 3y ago

> Carlisle got access to anonymized individual participant data (IPD) I'm not in the industry so my question might have an obvious answer to those of you who are: How would one go about getting IPD if you wanted to run your own analysis of trial data or other data-driven research?

misterdad 3y ago
You'll need to reach out to the study authors with a request. If they are interested (you're going to publish something noteworthy with a citation for them (low chance), you want to bring them in on some funded research (better chance), etc) then they'll push it to their Institutional Review Board (a group of usually faculty and sometimes administrative staff at a University/Hospital/Org) who will review the request, the conditions of the initial data collection, legal restrictions, and then decide if they'll proceed with setting up some sort of IRB agreement / data use agreement. Unless you're a tenured professor somewhere or a respected researcher with some outside group then you probably won't get past any of those steps. Even allegedly anonymized data comes with the risk of exposure (and real penalties) not to mention the administrative overhead (expense, time, attention) that you'll need to be able to cover the cost of through some funded research. That research, btw, will also need to be through some IRB structure. You can tap a private firm that acts as an IRB but that's another process entirely and most certainly requires fat stacks of cash. Legal privacy concerns, ethical concerns, careerism (nobody wants you to find the 'carry the two' you forgot so you can crash their career prospects), bloated expenses (somebody has to pay for all of that paperwork, all those IRB salaries, etc) and etc, etc, etc all keep reproducibility of individual data frozen. Even within the same institution. Within the same team! You have to tread lightly with reproduction.

deleted 3y ago

[deleted]

SubiculumCode 3y ago

I'd like to point out that the study [1] reported 16 trials from the USA with just one "zombie/fatally flawed" trial. e.g. 6% Most of the studies that were problematic came from China and Egypt. In other words, nothing new here. [1] https://associationofanaesthetists-publications.onlinelibrary.wiley.com/doi/10.1111/anae.15263 https://associationofanaesthetists-publications.onlinelibrar...

waterheater 3y ago

Two relevant perspectives to share: "It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of the New England Journal of Medicine." -Marcia Angell "If the image of medicine I have conveyed is one wherein medicine lurches along, riven by internal professional power struggles, impelled this way and that by arbitrary economic and sociopolitical forces, and sustained by bodies of myth and rhetoric that are elaborated in response to major threats to its survival, then that is the image supported by this study." -Evelleen Richards, "Vitamin C and Cancer: Medicine or Politics?"

avalys 3y ago
Ah, but don’t worry folks, those vaccines which generated $40B in profits for Pfizer and Moderna, those are just A-ok. Trust the science, which in this case is perfect, unassailable and uncorruptible (unlike everything else the pharma industry does). Don’t question the faith.
- chiefalchemist 3y ago
  And those weren't simple studies they were self-policed and self-reported trials. Trials that determined FDA green lighting. How can the FDA operate like that when the incentives to profit are 40B x more than being honest. File under: Follow the money
- strken 3y ago
  "Trust the science" is dumb, but "trust that the single-most reviewed medical studies of the past decade with 30k+ participants aren't outright bullshit" is enough for me. The risk for those studies is probably closer to what happened with Astrazeneca, where a very rare side effect got missed because 30k people weren't enough to detect it. I think skepticism is healthy to a degree, but ironically, the more adversarial the general public is to an area of study the more rigorous it usually is.
  - 3y ago

dimal 3y ago

I read those two percentages as not being added to each other. In other words, the 26% is a shittier subset of the 44%. At least that's what I think they mean.

toss1 3y ago

What I wonder is, since it is so easy for this one researcher to catch these flaws and lies, why are these not being caught by peer review? Increasing seems like peer review is a waste of everyone's time, and it's be better to just publishing when you think your ready on the likes of arXiv, and let everyone examine and criticize it. Run it up the flagpole, see if it gets saluted or shot down. Seems like there's be much more of an incentive to run confirmation studies.

ekianjo 3y ago

> It's quite difficult to imagine why studies Why? Because its just cargo cult science at the end of the day. Once you incentivize clinical trials to be used as a tool to unlock massive profits through an arbitrary process with a single regulator as judge it WILL be hacked. At every possible part of the process including regulators.

enkid 3y ago

Does the 44% not include the 26%?

eru 3y ago

> So, 70% fake/flawed. It's worse. First, there's a selection bias in what trials the author can get data for. Second, the 44% + 26% are just those where the author can detect problems from reading their data alone. If someone convincingly fakes their data, the author can't detect that.