6 ms·

In healthcare and life sciences research, I absolutely believe at least 80% is wrong and/or not reproducible. Not due to bad actors but, more likely, dataset b

In healthcare and life sciences research, I absolutely believe at least 80% is wrong and/or not reproducible. Not due to bad actors but, more likely, dataset bias or accidents upon patient matching algorithms or confounders. P < 0.05 is outdated at this point, for anything truly ground breaking p < 0.005 or < 0.0005 is probably a better choice, and even then I would ask "Where did you get your dataset from and did you combine (!!!) datasets from multiple orgs."

nradov 4y ago
Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. https://doi.org/10.1371/journal.pmed.0020124 https://doi.org/10.1371/journal.pmed.0020124
- epistasis 4y ago
  This paper would be more relevant if most science came down to evaluating a single p-value. Perhaps there are some fields of science where that is true, but I have not encountered them. I mostly look at molecular biology and infectious disease. Clinical trials in medicine come down to a single p-value by design, but don't fall prey to what he's talking about.
themitigating 4y ago
What is the source for your belief? Something unscientific perhaps?
- OrangeMonkey 4y ago
  The source for my beliefs is that I perform life sciences research to either generate new studies that are for publication or validate our studies against studies ran in the past. Its actually my job.
  - epistasis 4y ago
    Any study in life sciences will have lots of discussion of claims and modes for the future. But at the core will be the data, which is really the only thing that any researcher actually should trust from a paper. What are you designing new experiments for? Are you taking results in one cell like and condition and trying to transfer them to new cell lines and conditions? Are you trying to reproduce the same data in the same cel lines/organisms? There's such a huge wide array of what your could be taking about, but based on my experience in life sciences what you say does not seem the least bit realistic, in my first interpretation of what you are saying.
    - OrangeMonkey 4y ago
      The problem with getting more and more specific about my experiences and expertise is you get closer and closer to doxxing yourself. Honestly, it may be time to retire this account again. Also its worth noting that I am not claiming my experience is the end-all be-all. I am stating that, from my experience, I have incredible distrust for many studies that are published with 'amazing' results until peer reviewed preferably on disparate datasets. My experience is based around outcome based studies of the effect of drugs/treatment/regimens in oncology and oncology adjacent fields. This includes drugs treated alongside traditional cancer regimens to assist with managing adverse events and toxicities. Other fields may not have this reproducibility problem. Mine does. Even if the study design if perfect, and I can't imagine most are, the data itself can be questionable. Consider - what dataset would you use to identify if patients taking keytruda had a higher incidence of high blood pressure? You can use data from an EHR, licensed for deidentified studies, but EHR data is a burnt down trailer park of questionableness and its use in studies has been laughed at in many conferences. You can use data from individual enrolled patients (for a clinical study) but then the cost is extremely high vs a non interventional observational study using other data methods. The value of the data is likely to be higher, but since it costs more to collect maybe you are only in a few regions that may have a higher prevalence or incidence of this anyway. Troublesome. What about insurance data? You can get it cheaply, if you have high blood pressure GOOD doctors are likely to medicate you with a drug meant to treat it, and you can get it across the country. Seems good right? And it is, as you can generally draw an implication of high-blood-pressure->treatment-with-drug-x. So for a yes/no study it can help, but what if the base condition causes high blood pressure and we want to tell if the drug causes a HIGHER amount of high blood pressure than others. Insurance data by itself may not be enough to tell this data. So what do you do? You are stuck with no great answers.... and this is assuming your study design is perfect. So you buy multiple datasets in some third party health marketplace, and someone gets the great idea to combine the datasets to increase the n value. Well, too bad those datasets have a high overlap. So you have attributed a higher power to the study than is relevent. I hope this explains more about the concerns I have. Though, I suppose, it may mean that this account is now dead. I will have to think further. Anyway, hope you have a great day.
hackerlight 4y ago
P less than 0.0005 isn't much better because if that's the criteria then you still have to accept that psychic phenomena have been demonstrated in the lab. The methodology is deeply flawed, given that you haven't stopped accidental p-hacking (publication bias) and deliberate p-hacking (researcher data mining, e.g adding endpoints after looking at results), which compounds to create fake results with astronomically tiny p-values. You can only trust single, extremely large, canonical RCTs which were announced in advance, and in which you are confident there is no survivorship bias in terms of the possibility that this RCT would have been cancelled had the results been thought to be negative halfway through. Epidemiology (victim of researcher p-hacking and impossible to deal with confounders) or meta-analysis of RCTs or single small RCTs or RCTs that weren't announced in advance (victim of publication bias) should be taken with a grain of salt. If you accept conclusions drawn from these things at face value, be prepared to accept anything, because the methodology you've accepted is proven to be easily capable of demonstrating any fake phenomenon as true.
- OrangeMonkey 4y ago
  I think you hit it exactly right. There are so many ways that are studies can be flawed either accidentally or purposefully. I truly assume most studies are not purposefully fraudulent - people, by and large, are honest... but I do believe there are enough problems with our current methods that most studies are not truly accurate. Having a publish clinical trial regimen - created before your study starts - helps with a lot of this. It answers how you segment patients, match patients, handle patient dropout, and specifies what you are trying to compare for outcomes. Is it the end-all be-all? obviously not. I don't know what the true answer is.
ta988 4y ago
This is assuming your experiment was designed to test your hypothesis and that you model your null hypothesis properly. A p value means nothing anyway until you have both of these. Add to that the fact that many wrongly consider that a high p value disproves the original hypothesis.
- OrangeMonkey 4y ago
  I think you got it - there are many ways for studies to fail and its hard, at this time, to validate that a study succeeded except multiple studies confirming the original finding. Even then, I would worry that the results may be caused by some confounder in the original dataset/design instead of something you can trust.

nradov 4y ago

Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. https://doi.org/10.1371/journal.pmed.0020124 https://doi.org/10.1371/journal.pmed.0020124

epistasis 4y ago
This paper would be more relevant if most science came down to evaluating a single p-value. Perhaps there are some fields of science where that is true, but I have not encountered them. I mostly look at molecular biology and infectious disease. Clinical trials in medicine come down to a single p-value by design, but don't fall prey to what he's talking about.

themitigating 4y ago

What is the source for your belief? Something unscientific perhaps?

OrangeMonkey 4y ago
The source for my beliefs is that I perform life sciences research to either generate new studies that are for publication or validate our studies against studies ran in the past. Its actually my job.
- epistasis 4y ago
  Any study in life sciences will have lots of discussion of claims and modes for the future. But at the core will be the data, which is really the only thing that any researcher actually should trust from a paper. What are you designing new experiments for? Are you taking results in one cell like and condition and trying to transfer them to new cell lines and conditions? Are you trying to reproduce the same data in the same cel lines/organisms? There's such a huge wide array of what your could be taking about, but based on my experience in life sciences what you say does not seem the least bit realistic, in my first interpretation of what you are saying.
  - OrangeMonkey 4y ago
    The problem with getting more and more specific about my experiences and expertise is you get closer and closer to doxxing yourself. Honestly, it may be time to retire this account again. Also its worth noting that I am not claiming my experience is the end-all be-all. I am stating that, from my experience, I have incredible distrust for many studies that are published with 'amazing' results until peer reviewed preferably on disparate datasets. My experience is based around outcome based studies of the effect of drugs/treatment/regimens in oncology and oncology adjacent fields. This includes drugs treated alongside traditional cancer regimens to assist with managing adverse events and toxicities. Other fields may not have this reproducibility problem. Mine does. Even if the study design if perfect, and I can't imagine most are, the data itself can be questionable. Consider - what dataset would you use to identify if patients taking keytruda had a higher incidence of high blood pressure? You can use data from an EHR, licensed for deidentified studies, but EHR data is a burnt down trailer park of questionableness and its use in studies has been laughed at in many conferences. You can use data from individual enrolled patients (for a clinical study) but then the cost is extremely high vs a non interventional observational study using other data methods. The value of the data is likely to be higher, but since it costs more to collect maybe you are only in a few regions that may have a higher prevalence or incidence of this anyway. Troublesome. What about insurance data? You can get it cheaply, if you have high blood pressure GOOD doctors are likely to medicate you with a drug meant to treat it, and you can get it across the country. Seems good right? And it is, as you can generally draw an implication of high-blood-pressure->treatment-with-drug-x. So for a yes/no study it can help, but what if the base condition causes high blood pressure and we want to tell if the drug causes a HIGHER amount of high blood pressure than others. Insurance data by itself may not be enough to tell this data. So what do you do? You are stuck with no great answers.... and this is assuming your study design is perfect. So you buy multiple datasets in some third party health marketplace, and someone gets the great idea to combine the datasets to increase the n value. Well, too bad those datasets have a high overlap. So you have attributed a higher power to the study than is relevent. I hope this explains more about the concerns I have. Though, I suppose, it may mean that this account is now dead. I will have to think further. Anyway, hope you have a great day.

hackerlight 4y ago

P less than 0.0005 isn't much better because if that's the criteria then you still have to accept that psychic phenomena have been demonstrated in the lab. The methodology is deeply flawed, given that you haven't stopped accidental p-hacking (publication bias) and deliberate p-hacking (researcher data mining, e.g adding endpoints after looking at results), which compounds to create fake results with astronomically tiny p-values. You can only trust single, extremely large, canonical RCTs which were announced in advance, and in which you are confident there is no survivorship bias in terms of the possibility that this RCT would have been cancelled had the results been thought to be negative halfway through. Epidemiology (victim of researcher p-hacking and impossible to deal with confounders) or meta-analysis of RCTs or single small RCTs or RCTs that weren't announced in advance (victim of publication bias) should be taken with a grain of salt. If you accept conclusions drawn from these things at face value, be prepared to accept anything, because the methodology you've accepted is proven to be easily capable of demonstrating any fake phenomenon as true.

OrangeMonkey 4y ago
I think you hit it exactly right. There are so many ways that are studies can be flawed either accidentally or purposefully. I truly assume most studies are not purposefully fraudulent - people, by and large, are honest... but I do believe there are enough problems with our current methods that most studies are not truly accurate. Having a publish clinical trial regimen - created before your study starts - helps with a lot of this. It answers how you segment patients, match patients, handle patient dropout, and specifies what you are trying to compare for outcomes. Is it the end-all be-all? obviously not. I don't know what the true answer is.

ta988 4y ago

This is assuming your experiment was designed to test your hypothesis and that you model your null hypothesis properly. A p value means nothing anyway until you have both of these. Add to that the fact that many wrongly consider that a high p value disproves the original hypothesis.

OrangeMonkey 4y ago
I think you got it - there are many ways for studies to fail and its hard, at this time, to validate that a study succeeded except multiple studies confirming the original finding. Even then, I would worry that the results may be caused by some confounder in the original dataset/design instead of something you can trust.