3 ms·

Massive, major flaw in their argument. "without seeing the raw data" is not true. At least in the UK. All raw study data, published and un published must be pr

Massive, major flaw in their argument. "without seeing the raw data" is not true. At least in the UK. All raw study data, published and un published must be provided before a medicine will be reimbursed. It is reviewed by teams who do their utmost to prove the med is worthless. Same for the FDA afaik. All the data is generally "commercial in confidence" so any old twitteraty can not download and misinterpret it. All trials must be registered on Clinicaltrials.gov or they cannot be used as evidence. Publication bias is a well know issue, resolved for more than a decade. No idea how this article made it into the BMJ, smells like click bait.

rscho 5y ago
> Publication bias is a well know issue, resolved for more than a decade. How is that resolved? This is not what I see in my meta-analyses.
- DarkmSparks 5y ago
  requiring registering with clinicaltrials.gov and regulators doing their own analysis on all the data. Managed entry agreements and conditional reimbursement (dont pay for patients that dont do better than the average of current practice) Requirements for "real world data" collection and submission post approval. A whole host of other mechanisms I cant remember off the top of my head. Also "evidence based medicine" doesnt mean the evidence is infallible (e.g. rare disease meds have a huge issue with collecting any meaningful data) - it just means actually using evidence rather than just taking the industries and acedemias word for it that it works like the old days.
  - rscho 5y ago
    From personal experience, I think you are massively overestimating the reach and effectiveness of regulations.
    - DarkmSparks 5y ago
      I've worked with nice. several of my students still do. We've kicked out of the NHS quite a few meds the industry and academics were wrong about, based on the evidence, not what the industry/academics said before they were shown to be not correct. The whole posit of this article is wrong. "acedemics said so" is not evidence based medicine, it is one small facit of it, and the author matters, no one trusts "sponsored by" papers, NIHR is constantly publishing counter evidence. Some of that even ends up in court (e.g. Avastin) https://www.reuters.com/article/us-novartis-bayer-britain-idUSKCN1M12EC https://www.reuters.com/article/us-novartis-bayer-britain-id...
      - rscho 5y ago
        This I totally agree with. I've not much experience with the UK system, but I disagree that publication bias is solved, at least regarding the scientific literature at large.
        DarkmSparks 5y ago
        Well. I guess you could probably pin some blame on publication bias for the massive misinvestment that went into the recent dementia meds that all failed at phase 3. Something made them get the science badly wrong. I was talking specifically in the context of evidence based medicine for regulatory decision making, which is the process that replaced blue envelopes full of money or throwing acedemic papers at an important politician whose relative had the disease that came before it.
  - Brakenshire 5y ago
    Do these issues not still apply? Have they been fixed? https://en.m.wikipedia.org/wiki/AllTrials https://en.m.wikipedia.org/wiki/AllTrials
    - DarkmSparks 5y ago
      They are "fixed" by extending the evidence base for decision making well beyond raw clinical trial data and publications on them (none of which ever translates into real world practice outcomes anyway). In important instances (such as expensive meds going to large numbers of people) the payer will commission their own research e.g. through NIHR https://www.nihr.ac.uk/ https://www.nihr.ac.uk/ There is a whole field of "value of information" now to estimate how incorrect information would affect a decision and whether it warrants research investment (which covers publication bias - e.g. how many unpublished studies showing no or negative effects would there need to be to change the decision)
radford-neal 5y ago
"All the data is generally "commercial in confidence" so any old twitteraty can not download and misinterpret it." This is totally the wrong attitude. Everyone should be able to look at the data. The established authorities are absolutely capable of totally misinterpreting data. The only hope is to let other people look at it.
- DarkmSparks 5y ago
  In theory I agree with you. In practice, Ive seen too many bad faith actors intentionally drawing false conclusions and then linking to data they say agrees with them that clearly doesnt. Twitter, wikipedia and facebook are terrible for this behaviour, many of the so called fact checkers just as bad. Healthcare trial data analysis is a highly specialised skill, (last one I did took about 3 months of meetings just to understand the dataset, about 6000 columns of abbreviations of complex medical terms) those with the qualifications to analyse it are typically not refused access if they actually want it, something like GPRD is generally more valuable and available to everyone. In Sweden everyones entire medical history is open access for research. There isnt any shortage of data, there is an acute shortage of analysts and finance to cover the costs involved.
  - radford-neal 5y ago
    I have seen many studies where they say that qualified investigators can have access to the data if they sign lots of forms, agree to collaborate with (not just acknowledge) the original authors, and wait six months for this all to be approved. I expect that I would be approved if I did all this, but it's not worth it. In some cases I suspect the study is flawed, but debunking it months later would be a lot of work for little return. In other cases, the study looks OK, but without ready access to the data to confirm this (and perhaps get more information with a more sophisticated analysis), it's really pretty much worthless. Yes, worthless. As in no one should put any weight on such unconfirmable "evidence". It's just wasted effort by the investigators, who piddled away their time (and their funding agency's money) because they're not willing to release the data. These are not studies where one would need 3 months to just understand what the variable names mean - perhaps the data was extracted from such a huge database, and perhaps there are issues with how that was done, but there are also quite possibly issues with what they did afterwards, which can be examined separately. Comments by unqualified people on twitter are a fact of life. You can't get rid of them without also eliminating comments that reveal mistakes (even fraud) in what superficially seems like authoritative research. Your judgement that such uninformed comments are intentional lies is probably untrue in most cases.

rscho 5y ago

> Publication bias is a well know issue, resolved for more than a decade. How is that resolved? This is not what I see in my meta-analyses.

DarkmSparks 5y ago
requiring registering with clinicaltrials.gov and regulators doing their own analysis on all the data. Managed entry agreements and conditional reimbursement (dont pay for patients that dont do better than the average of current practice) Requirements for "real world data" collection and submission post approval. A whole host of other mechanisms I cant remember off the top of my head. Also "evidence based medicine" doesnt mean the evidence is infallible (e.g. rare disease meds have a huge issue with collecting any meaningful data) - it just means actually using evidence rather than just taking the industries and acedemias word for it that it works like the old days.
- rscho 5y ago
  From personal experience, I think you are massively overestimating the reach and effectiveness of regulations.
  - DarkmSparks 5y ago
    I've worked with nice. several of my students still do. We've kicked out of the NHS quite a few meds the industry and academics were wrong about, based on the evidence, not what the industry/academics said before they were shown to be not correct. The whole posit of this article is wrong. "acedemics said so" is not evidence based medicine, it is one small facit of it, and the author matters, no one trusts "sponsored by" papers, NIHR is constantly publishing counter evidence. Some of that even ends up in court (e.g. Avastin) https://www.reuters.com/article/us-novartis-bayer-britain-idUSKCN1M12EC https://www.reuters.com/article/us-novartis-bayer-britain-id...
    - rscho 5y ago
      This I totally agree with. I've not much experience with the UK system, but I disagree that publication bias is solved, at least regarding the scientific literature at large.
      - DarkmSparks 5y ago
        Well. I guess you could probably pin some blame on publication bias for the massive misinvestment that went into the recent dementia meds that all failed at phase 3. Something made them get the science badly wrong. I was talking specifically in the context of evidence based medicine for regulatory decision making, which is the process that replaced blue envelopes full of money or throwing acedemic papers at an important politician whose relative had the disease that came before it.
- Brakenshire 5y ago
  Do these issues not still apply? Have they been fixed? https://en.m.wikipedia.org/wiki/AllTrials https://en.m.wikipedia.org/wiki/AllTrials
  - DarkmSparks 5y ago
    They are "fixed" by extending the evidence base for decision making well beyond raw clinical trial data and publications on them (none of which ever translates into real world practice outcomes anyway). In important instances (such as expensive meds going to large numbers of people) the payer will commission their own research e.g. through NIHR https://www.nihr.ac.uk/ https://www.nihr.ac.uk/ There is a whole field of "value of information" now to estimate how incorrect information would affect a decision and whether it warrants research investment (which covers publication bias - e.g. how many unpublished studies showing no or negative effects would there need to be to change the decision)

radford-neal 5y ago

"All the data is generally "commercial in confidence" so any old twitteraty can not download and misinterpret it." This is totally the wrong attitude. Everyone should be able to look at the data. The established authorities are absolutely capable of totally misinterpreting data. The only hope is to let other people look at it.

DarkmSparks 5y ago
In theory I agree with you. In practice, Ive seen too many bad faith actors intentionally drawing false conclusions and then linking to data they say agrees with them that clearly doesnt. Twitter, wikipedia and facebook are terrible for this behaviour, many of the so called fact checkers just as bad. Healthcare trial data analysis is a highly specialised skill, (last one I did took about 3 months of meetings just to understand the dataset, about 6000 columns of abbreviations of complex medical terms) those with the qualifications to analyse it are typically not refused access if they actually want it, something like GPRD is generally more valuable and available to everyone. In Sweden everyones entire medical history is open access for research. There isnt any shortage of data, there is an acute shortage of analysts and finance to cover the costs involved.
- radford-neal 5y ago
  I have seen many studies where they say that qualified investigators can have access to the data if they sign lots of forms, agree to collaborate with (not just acknowledge) the original authors, and wait six months for this all to be approved. I expect that I would be approved if I did all this, but it's not worth it. In some cases I suspect the study is flawed, but debunking it months later would be a lot of work for little return. In other cases, the study looks OK, but without ready access to the data to confirm this (and perhaps get more information with a more sophisticated analysis), it's really pretty much worthless. Yes, worthless. As in no one should put any weight on such unconfirmable "evidence". It's just wasted effort by the investigators, who piddled away their time (and their funding agency's money) because they're not willing to release the data. These are not studies where one would need 3 months to just understand what the variable names mean - perhaps the data was extracted from such a huge database, and perhaps there are issues with how that was done, but there are also quite possibly issues with what they did afterwards, which can be examined separately. Comments by unqualified people on twitter are a fact of life. You can't get rid of them without also eliminating comments that reveal mistakes (even fraud) in what superficially seems like authoritative research. Your judgement that such uninformed comments are intentional lies is probably untrue in most cases.