4 ms·

Meta-analysis have a good place. But they also have known flaws/assumptions. Often it is impractical to perform large studies. Partially by logistics and often

Meta-analysis have a good place. But they also have known flaws/assumptions. Often it is impractical to perform large studies. Partially by logistics and often by funding. But if there are lots of smaller studies you can aggregate the data to check for overall significant results. But mainly they are helpful to give a "survey" of the current research instead of having to link to 10 different studies and hope someone else sorts out what the data says. But as you point out, there are two major flaws in the assumptions. First, that the scientific procedure is sound. Secondly, that the data is handled properly, and thus you can take the summary and back out the underlying data. Unfortunately trying to fix the first is really, really hard. The second is somewhat mitigatable. As the Nature article suggests, you could publish the underlying data (anonymized of course). This would help in two ways. First, the meta-analysis could check for confounding variables to control across all of the data. The second major one is it would help people spot fraudulent data. However, as anyone who handles datasets knows, publishing and wrangling data into a useable state from multiple sources is a serious pain in the neck. Plus a lot of concerns about how de-anonymized the data would be. As we've known, with enough metadata it can be used to identify individuals. And publicly publishing dais data would definitely allow for some serious sleuthing work to be done.

trts 5y ago
The problem I saw with the Ivermectin metastudies (which I did a fair amount of research on given that people close to me had decided to employ it as a prophylactic against Covid) was that the people who did the metanalyses were trained to read the results rather than grapple with the methodology and statistics. None of the study results were robust, but many of the study outcomes were positive. So they were counting up positive outcomes and saying that more often than not, it was evidence that Ivermectin was effective. This is a good means of generating a hypothesis, but a bad means of generating a conclusion. Most of these studies were not in any way controlled or comparable in their conditions, and some had a very low number of observations. It didn't seem plausible that the data could be aggregated together. It will be very interesting to see the results of the larger RCTs underway. Regardless of the outcome there _has_ to be a lesson for at least some people who arrived at fervent conclusions about what a miracle/scam this particular drug is. Unfortunately it seems unlikely that it will be a lesson that endures.
- dboreham 5y ago
  In my mind, there is also underlying all this the idea that if a drug actually works, then there will be many doctors going on CNN to say that it totally works; that they're super excited by their patients recovering. In other words, the effect should be profound. If the effect is marginal, why bother, even if it is a real measurable effect in well conducted trials? I mean : that doctors observe that some therapy or drug works well is step 1, then step 2 is to conduct trials to prove conclusively that step 1 wasn't a statistical fluke. In this case we seem to have it backwards : we first observed that there is an effect in vitro, then ran trials which proved nothing much, then extracted noise from that data to prove whatever we wanted.
  - colordrops 5y ago
    Determining effectiveness through the number or media reach of doctors talking about it is the opposite of science.
  - bb88 5y ago
    > ...then there will be many doctors going on CNN to say that it totally works... I don't think we can trust "doctors" anymore. There were a bunch of doctors (aka America's Frontline Doctors) that were trying to go around and say Ivermectin works. Same with HCQ. The "needle rape" doctor in Idaho said the vaccine is dangerous, but Ivermectin saves lives. In Idaho, I assure you, doctor's are prescribing patients Ivermectin in the ICU. If it really did work Boise would be the groundswell of Ivermectin miracles, where people are walking out of the ICU, ending the crisis. Unfortunately, that's not happening. ICU's are primarily being freed up from death now it seems like. ICU's are in the high 90's capacity. https://idahocapitalsun.com/2021/09/22/today-in-idaho-hospitals-and-covid-19-patients-er-visits-and-capacity/ https://idahocapitalsun.com/2021/09/22/today-in-idaho-hospit...
  - Fjolsvith 5y ago
    This might be true except for the fact that news outlets follow a script to push a narrative (and CNN is not exempted from this): https://www.youtube.com/watch?v=ksb3KD6DfSI https://www.youtube.com/watch?v=ksb3KD6DfSI
rscho 5y ago
I've published (and still am publishing) multiple meta analyses. IMO,meta analysis on aggregate data is an h-index gaming machine that can never increase the value of its constituents. The amount of missing info in aggregate data is such that it's _always_ impossible to know if the constituent studies really draw from the same population, which is the cornerstone of meta analysis. And that's very practical, because most authors (myself included) mainly use meta analysis as a means to build a career, and not at all for doing science. IMO, current clinical research has very little science left in it. That's a huge shame, and it's a real priority to make that change!
nradov 5y ago
The federal government has specific guidance on how to meet the legal criteria for de-identifying clinical data. Once researchers meet that standard they're generally free to publish raw data, unless there's a stricter standard imposed by state law or some sort of institutional review board. https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html https://www.hhs.gov/hipaa/for-professionals/privacy/special-...
aeternum 5y ago
What do you think about the ability for researches to pick & choose which papers to include in a meta-analysis? Double-blind is important for a reason, researchers are exceedingly good at proving what they want to prove even if subconscious, can meta-analysis studies ever overcome this?

trts 5y ago

The problem I saw with the Ivermectin metastudies (which I did a fair amount of research on given that people close to me had decided to employ it as a prophylactic against Covid) was that the people who did the metanalyses were trained to read the results rather than grapple with the methodology and statistics. None of the study results were robust, but many of the study outcomes were positive. So they were counting up positive outcomes and saying that more often than not, it was evidence that Ivermectin was effective. This is a good means of generating a hypothesis, but a bad means of generating a conclusion. Most of these studies were not in any way controlled or comparable in their conditions, and some had a very low number of observations. It didn't seem plausible that the data could be aggregated together. It will be very interesting to see the results of the larger RCTs underway. Regardless of the outcome there _has_ to be a lesson for at least some people who arrived at fervent conclusions about what a miracle/scam this particular drug is. Unfortunately it seems unlikely that it will be a lesson that endures.

dboreham 5y ago
In my mind, there is also underlying all this the idea that if a drug actually works, then there will be many doctors going on CNN to say that it totally works; that they're super excited by their patients recovering. In other words, the effect should be profound. If the effect is marginal, why bother, even if it is a real measurable effect in well conducted trials? I mean : that doctors observe that some therapy or drug works well is step 1, then step 2 is to conduct trials to prove conclusively that step 1 wasn't a statistical fluke. In this case we seem to have it backwards : we first observed that there is an effect in vitro, then ran trials which proved nothing much, then extracted noise from that data to prove whatever we wanted.
- colordrops 5y ago
  Determining effectiveness through the number or media reach of doctors talking about it is the opposite of science.
- bb88 5y ago
  > ...then there will be many doctors going on CNN to say that it totally works... I don't think we can trust "doctors" anymore. There were a bunch of doctors (aka America's Frontline Doctors) that were trying to go around and say Ivermectin works. Same with HCQ. The "needle rape" doctor in Idaho said the vaccine is dangerous, but Ivermectin saves lives. In Idaho, I assure you, doctor's are prescribing patients Ivermectin in the ICU. If it really did work Boise would be the groundswell of Ivermectin miracles, where people are walking out of the ICU, ending the crisis. Unfortunately, that's not happening. ICU's are primarily being freed up from death now it seems like. ICU's are in the high 90's capacity. https://idahocapitalsun.com/2021/09/22/today-in-idaho-hospitals-and-covid-19-patients-er-visits-and-capacity/ https://idahocapitalsun.com/2021/09/22/today-in-idaho-hospit...
- Fjolsvith 5y ago
  This might be true except for the fact that news outlets follow a script to push a narrative (and CNN is not exempted from this): https://www.youtube.com/watch?v=ksb3KD6DfSI https://www.youtube.com/watch?v=ksb3KD6DfSI

rscho 5y ago

I've published (and still am publishing) multiple meta analyses. IMO,meta analysis on aggregate data is an h-index gaming machine that can never increase the value of its constituents. The amount of missing info in aggregate data is such that it's _always_ impossible to know if the constituent studies really draw from the same population, which is the cornerstone of meta analysis. And that's very practical, because most authors (myself included) mainly use meta analysis as a means to build a career, and not at all for doing science. IMO, current clinical research has very little science left in it. That's a huge shame, and it's a real priority to make that change!

nradov 5y ago

The federal government has specific guidance on how to meet the legal criteria for de-identifying clinical data. Once researchers meet that standard they're generally free to publish raw data, unless there's a stricter standard imposed by state law or some sort of institutional review board. https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html https://www.hhs.gov/hipaa/for-professionals/privacy/special-...

aeternum 5y ago

What do you think about the ability for researches to pick & choose which papers to include in a meta-analysis? Double-blind is important for a reason, researchers are exceedingly good at proving what they want to prove even if subconscious, can meta-analysis studies ever overcome this?