5 ms·

This has interesting implications to the recent discussions about A/B testing and Multi-armed bandit algorithms. Assuming double blind tests are akin to A/B te

This has interesting implications to the recent discussions about A/B testing and Multi-armed bandit algorithms. Assuming double blind tests are akin to A/B testing, at what point was the trial stopped? Was it stopped before there was statistical significance? Or, are there differences in how double blind tests are done than how A/B testing is done in software and websites?

deleted 14y ago
[deleted]
- refurb 14y ago
  What do you mean by triple blinded? The research was double-blinded, what more are you looking for?
  - deleted 14y ago
    [deleted]
  - saurik 14y ago
    http://medical-dictionary.thefreedictionary.com/_/dict.aspx?word=triple+blind http://medical-dictionary.thefreedictionary.com/_/dict.aspx?...
    - refurb 14y ago
      OK, but what makes you think this wasn't triple-blinded? I've never heard a clinical trial called that, but usually the data analysts don't know which patients got the drug either.
planetguy 14y ago
at what point was the trial stopped? Was it stopped before there was statistical significance? I'm guessing... no. I'm guessing they don't call off a zillion-dollar phase III clinical cancer drug trial without running it by someone who has glanced at an undergrad statistics textbook. We're not just banging rocks together here, guys, we know how to do clinical trials.
- DanBC2 14y ago
  > We're not just banging rocks together here, guys, we know how to do clinical trials. There are many problems with clinical trials. Read Ben Goldacre for examples. One problem is that good results are published, while bad results (when they're submitted, which isn't often) are too boring for publication. And then there are all the other problems of studies that aren't correctly blinded, or the selection is faulty, or etc. These are all reputable pharma companies with substantial funds to run tests. I am, obviously, addressing your comment because there's not nearly enough detail in the article to discuss.
  - refurb 14y ago
    Often bad clinical trial results aren't released because they are bad for reasons other than the drug doesn't work. Also, it's very rare for big pharma companies to screw up a clinical trial. It's much more common in the small biotechs and non-profits. If there is one thing that big pharma companies are good at is drug development (not so much drug research).
    - DanBC2 14y ago
      So a big pharma company like Merck wouldn't use a big publishing company like Elsevier to create fake "medical" journals and publish research? https://www.google.co.uk/search?&q=elseviers+fake+journal+merck https://www.google.co.uk/search?&q=elseviers+fake+journa...
      - streptomycin 14y ago
        What does that have to do with clinical trials? You're not being logically coherent. The FDA isn't going to approve a drug because someone makes a fake journal and publishes some results in it.
  - wisty 14y ago
    > There are many problems with clinical trials. Read Ben Goldacre for examples Yes, and the statisticians in the FDA and the drug companies have a lot of experience in this. I think they started looking at ways to end trials early back in the ADT (anti-HIV drug) days. It was partly due to popular pressure - people with AIDS didn't want to wait for the clinical trials. Lots of big angry protests. And the cat and mouse game between regulators and big pharma cooking the books is also pretty mature. There's problems with the system, but bad statistics is one of the smaller ones. The real problem is marginally effective drugs being used instead of better patent-expired ones, because the less effective (but not truly bad - they beat placebo) ones have bigger marketing budgets.
jerf 14y ago
We are hearing about this precisely because it is unusual and thus newsworthy. This article is light on the scientific details. Given those things, let's extend them the benefit of the doubt for now and hold the standard (and somewhat tedious) litany of complaints until there's more evidence that they are called for, as exceptional actions in such a tightly regulated domain is reasonably likely to imply exceptional circumstances.
Mvandenbergh 14y ago
No. The effect size was large enough that the statistical significance agreed with the FDA before the start of the study was reached well before the end of the trial. This is not uncommon, and the possibility of ending the trial early will have been decided beforehand. Usually an independent advisory board not affiliated with the FDA or the pharma company will make these decisions - the other parties may not even have known that the results were better than expected until the independent board decided to end the study.
Gatsky 14y ago
I was at the announcement of the trial data by the investigators this morning. The results were also discussed by a statistician not involved with the trial directly. Not sure that hacker news needs an in depth analysis of the trial design and outcome, but anyway... In summary, the trial looked at patients who had incurable prostate cancer that had spread and stopped responding to traditional hormonal treatments. Patients had to have no symptoms to be enrolled on this trial. The trial compared abiraterone, a new type of hormonal treatment plus steroids to just being on steroids alone. The question that the trial was asking can be framed as "Does taking abiraterone prolong the period of time that a patient can be symptom free and their cancer is under control?" That was the primary endpoint. A secondary endpoint was survival, or how long they lived. Built in to the study design was a planned interim analysis of the data. Normally, the investigators are not allowed to just look at the data as it accrues continuously because it creates bias. It is a standard approach to allow one look at the data when say 40% of the expected events (in this case, development of progressive cancer) have occurred. When they did this, the difference between getting the drug and not getting the drug was considered big enough to unblind the trial. This means that everyone finds out if they were getting placebo or the real drug. There are multiple reasons why one may decide to do this. First and foremost is that if the drug really works, it would be unethical to leave people on a placebo for long periods, because it is clear they have an inferior outcome. Stopping early also speeds the time to drug approval and market entry for a given indication. There are somewhat esoteric statistical procedures that can determine at an interim analysis if the accrued data shows a statistically significant difference between groups, which would therefore justify stopping the trial, while adjusting for the impact of repeated testing on the type I error rate (ie the chance of finding an association that doesn't really exist). In this trial, there is no doubt that the primary endpoint was positive and clinically and statistically significant, even with only 40% of people on the trial having had a progression event. So they stopped. The data on whether abiraterone reduces the chance of death are not mature, because not enough of the people on the trial have died yet, so those results will be awaited. Stopping early means that these results may be biased however, because the trial is no longer blinded. This was a large and well done trial, that employed standard statistical practices.
- firebones 14y ago
m104 14y ago
No interesting implications, just the reaffirmation that real significance is more important than statistical significance.

deleted 14y ago

[deleted]

refurb 14y ago
What do you mean by triple blinded? The research was double-blinded, what more are you looking for?
- deleted 14y ago
  [deleted]
- saurik 14y ago
  http://medical-dictionary.thefreedictionary.com/_/dict.aspx?word=triple+blind http://medical-dictionary.thefreedictionary.com/_/dict.aspx?...
  - refurb 14y ago
    OK, but what makes you think this wasn't triple-blinded? I've never heard a clinical trial called that, but usually the data analysts don't know which patients got the drug either.

planetguy 14y ago

at what point was the trial stopped? Was it stopped before there was statistical significance? I'm guessing... no. I'm guessing they don't call off a zillion-dollar phase III clinical cancer drug trial without running it by someone who has glanced at an undergrad statistics textbook. We're not just banging rocks together here, guys, we know how to do clinical trials.

DanBC2 14y ago
> We're not just banging rocks together here, guys, we know how to do clinical trials. There are many problems with clinical trials. Read Ben Goldacre for examples. One problem is that good results are published, while bad results (when they're submitted, which isn't often) are too boring for publication. And then there are all the other problems of studies that aren't correctly blinded, or the selection is faulty, or etc. These are all reputable pharma companies with substantial funds to run tests. I am, obviously, addressing your comment because there's not nearly enough detail in the article to discuss.
- refurb 14y ago
  Often bad clinical trial results aren't released because they are bad for reasons other than the drug doesn't work. Also, it's very rare for big pharma companies to screw up a clinical trial. It's much more common in the small biotechs and non-profits. If there is one thing that big pharma companies are good at is drug development (not so much drug research).
  - DanBC2 14y ago
    So a big pharma company like Merck wouldn't use a big publishing company like Elsevier to create fake "medical" journals and publish research? https://www.google.co.uk/search?&q=elseviers+fake+journal+merck https://www.google.co.uk/search?&q=elseviers+fake+journa...
    - streptomycin 14y ago
      What does that have to do with clinical trials? You're not being logically coherent. The FDA isn't going to approve a drug because someone makes a fake journal and publishes some results in it.
- wisty 14y ago
  > There are many problems with clinical trials. Read Ben Goldacre for examples Yes, and the statisticians in the FDA and the drug companies have a lot of experience in this. I think they started looking at ways to end trials early back in the ADT (anti-HIV drug) days. It was partly due to popular pressure - people with AIDS didn't want to wait for the clinical trials. Lots of big angry protests. And the cat and mouse game between regulators and big pharma cooking the books is also pretty mature. There's problems with the system, but bad statistics is one of the smaller ones. The real problem is marginally effective drugs being used instead of better patent-expired ones, because the less effective (but not truly bad - they beat placebo) ones have bigger marketing budgets.

jerf 14y ago

We are hearing about this precisely because it is unusual and thus newsworthy. This article is light on the scientific details. Given those things, let's extend them the benefit of the doubt for now and hold the standard (and somewhat tedious) litany of complaints until there's more evidence that they are called for, as exceptional actions in such a tightly regulated domain is reasonably likely to imply exceptional circumstances.

Mvandenbergh 14y ago

No. The effect size was large enough that the statistical significance agreed with the FDA before the start of the study was reached well before the end of the trial. This is not uncommon, and the possibility of ending the trial early will have been decided beforehand. Usually an independent advisory board not affiliated with the FDA or the pharma company will make these decisions - the other parties may not even have known that the results were better than expected until the independent board decided to end the study.

Gatsky 14y ago

I was at the announcement of the trial data by the investigators this morning. The results were also discussed by a statistician not involved with the trial directly. Not sure that hacker news needs an in depth analysis of the trial design and outcome, but anyway... In summary, the trial looked at patients who had incurable prostate cancer that had spread and stopped responding to traditional hormonal treatments. Patients had to have no symptoms to be enrolled on this trial. The trial compared abiraterone, a new type of hormonal treatment plus steroids to just being on steroids alone. The question that the trial was asking can be framed as "Does taking abiraterone prolong the period of time that a patient can be symptom free and their cancer is under control?" That was the primary endpoint. A secondary endpoint was survival, or how long they lived. Built in to the study design was a planned interim analysis of the data. Normally, the investigators are not allowed to just look at the data as it accrues continuously because it creates bias. It is a standard approach to allow one look at the data when say 40% of the expected events (in this case, development of progressive cancer) have occurred. When they did this, the difference between getting the drug and not getting the drug was considered big enough to unblind the trial. This means that everyone finds out if they were getting placebo or the real drug. There are multiple reasons why one may decide to do this. First and foremost is that if the drug really works, it would be unethical to leave people on a placebo for long periods, because it is clear they have an inferior outcome. Stopping early also speeds the time to drug approval and market entry for a given indication. There are somewhat esoteric statistical procedures that can determine at an interim analysis if the accrued data shows a statistically significant difference between groups, which would therefore justify stopping the trial, while adjusting for the impact of repeated testing on the type I error rate (ie the chance of finding an association that doesn't really exist). In this trial, there is no doubt that the primary endpoint was positive and clinically and statistically significant, even with only 40% of people on the trial having had a progression event. So they stopped. The data on whether abiraterone reduces the chance of death are not mature, because not enough of the people on the trial have died yet, so those results will be awaited. Stopping early means that these results may be biased however, because the trial is no longer blinded. This was a large and well done trial, that employed standard statistical practices.

firebones 14y ago

m104 14y ago

No interesting implications, just the reaffirmation that real significance is more important than statistical significance.