9 ms·

Why Are Clinical Trials So Complicated?

106 points by yread 7y ago

Anthony-G 7y ago
For someone who never looked into the complexities of clinical trials, I thought that was a detailed and informative answer. It seems the key point to take away is, “The amount of noise in human clinical data can defy belief if you haven’t seen it in person”. :)
- pfdietz 7y ago
  A truth worth repeating!
- zxexz 7y ago
  I’ve been working in this space for a while now, and this still blows my mind. Patient-reported data is noisy, doctor’s notes infamously confusing and inconsistent in formatting. Even fields you can pull from the health system’s EHR directly will be in disparate formats across different hospitals, or even different departments. It would be easy to say it all should be standardized (and it should, for the most part), but workflow considerations keep standards from being adopted. What a data standard for oncology data would look like is very different from what one developed for ophthalmology. Not to mention UX - if you’re a doctor seeing 20 patients a day, an extra 15 minutes of data entry per patient could bring that number down to 15 patients or less. UX and data standards really does have a direct link to patient outcomes in healthcare!
  - yread 7y ago
    Indeed, we had to recently work with 500k patients' appointments. Database table Appointment had a date and "actually met" flag - sounds simple enough, right? Nope - doctors were "meeting" dead people, different departments had different rules about whether phone calls were included and what did the "actually met" flag actually mean.
antipaul 7y ago
The saying in our industry is, "clinical trials are the most complicated thing in the world". EDIT: adding details: You have to mobilize large interdisciplinary teams to come up with products that are inherently very hard to invent, and which, after all that, can easily fail a trail and never see light of day, or can then still harm or kill people even after approved, all while operating in an understandbly extremely regulated and complicated environment. One example: One of the many piles of documents that you need to submit to a national health authority _just so they consider your product_ (before you can release to a single "user"!), is thorough documentation of how you _plan_ to manufacture the product, if it were approved. Just these text files can weigh _gigabytes_. Think how much effort it would take just to write something coherent that is so long! And of course, you have to get other teams to run a bunch of experiments to inform what to write. And this is even before you are actually producing the product for release. Oh, and maybe your product won't actually be approved, so all that goes to waste. And that's just the beginning. Because all of that excludes _any_ mention of the real deal, which is whether the product actually works or is safe. All of that requires its own experiments and documentation. And all that assumes that decades of prior research actually led to some useful prototype where you can start. And all that excludes any mention of ethical considerations like who pays for the research, what is a just price for a cancer drug - heck, whether you should even be trying to make a profit off of lifesaving drugs. If getting to a point like selling your app called Whats App for $18 billion (or whatever it was) would require a walk down the street, then getting a drug to market is like leaving the earth, flying past the furthest planets, and then trying to reach another galaxy.
- Madmallard 7y ago
  Why do I have a hard-time believing this? So many really bad drugs that are popularized and thrown with black-box warnings and such after the fact. I think the integrity of that system has been subverted.
  - lobotryas 7y ago
    The key to taking a drug through trial is having enough cash for the trial and for your company to survive the time it takes to complete the trial successfully. Remember that said success is not guaranteed and you might have to write off years of work and start from scratch. This is why drugs are so expensive in the West.
Tomte 7y ago
> Once in a while, they’ll even say “Stop right now – your drug is so damn good that it’s unethical not to give it to everyone in the trial”, but years go by when that does not happen, anywhere in the industry. We've had such a case a short time ago (months?) in the news. Does anyone remember what that was about or even has a link?
- alexrson 7y ago
  You may be thinking of Tagrisso: https://www.statnews.com/2020/04/10/astrazeneca-tagrisso-study-ended-overwhelming-benefit/ https://www.statnews.com/2020/04/10/astrazeneca-tagrisso-stu...
- antipaul 7y ago
  Just because it's not reported in the Times, doesn't mean it doesn't happen. It actually can happen quite frequently. Trial protocols can pre-specify early stopping criteria: if we see drug A is better than drug B by a factor of X at time T, then we can declare superiority with given statistical confidence.
- zxexz 7y ago
  A kind of side-note, but for similar reasons, most interventional trials in the oncology space aren’t double-blinded for that reason. If you have cancer and sign up for a trial, you’ve likely run out of standard treatment options, so being given a placebo in a potentially life-saving therapy is not easy to justify. The different arms of the trial tend to vary in inclusion/exclusion criteria and doses, but only very rarely is there a placebo involved.
  - tialaramex 7y ago
    Yup. I had Hodgkins many years ago. We've been treating Hodgkins for many decades with considerable success (~95% of young men with Hodgkins eventually die of something else and that may have improved further). And so it would be unethical to perform a placebo comparison because we're confident from anecdotal evidence (some patients refuse treatment) that it'll kill you. I was considered for (but ultimately did not qualify IIRC) a trial of a new Hodgkins treatment. If I had qualified and opted in all that would have meant is basically they toss a coin, heads I get the new treatment, tails I get the existing "Gold standard". As well as not using a placebo because it's a death sentence, they also can't truly blind the trial because the new regime is different drugs, with different side effects and a different calendar. The staff must know which they are giving you and even if they didn't just tell you it's not hard to look up online what the Gold Standard is and tell if you aren't getting it. So all they have left it's a two arm trial in which they've tried to match the arms up, and then compare the results. In my case the eventual trial results show the new regime was slightly worse, not better as hoped. Slightly more (statistically insignificant but not good news) deaths and worse side effects. Unfortunate, but there was only one way to find out.
choxi 7y ago
> The amount of noise in human clinical data can defy belief if you haven’t seen it in person I'm not surprised that patient compliance is a problem, but it still seems like an expensive burden to find patients physically near the clinical trials so they can come in to take the medication. It just seems like a logistical nightmare that software could help with. Could you build an app that requires patients to record themselves taking their medication under whatever conditions they're supposed to? You might have to manually audit all of the recordings, but the number of people that still fail to comply might be offset but how many more patients you can include in the study. There might be a lot of other things you could verify remotely using wearable tech, e.g. heart rate now that Apple's EKG monitoring is FDA approved. Great read, I'd like to see more "Chesterton's Fence" arguments for other highly regulated industries.
- hpaavola 7y ago
  "Could you build an app that requires patients to record themselves taking their medication under whatever conditions they're supposed to?" Yes. I work for a company (Signant Health) which does clinical trial platform as a service (among other things) and our platform has this feature that the trials can have steps which require the users to video certain things. And we are hiring https://signanthealth.com/careers/ https://signanthealth.com/careers/ :)
- doctoring 7y ago
  Many trials do NOT require patients to come in to take the medication, but there are of course some which are harder to take at home (whether intravenous or intrathecal or whatever). But compliance is certainly hard to assess and so important. Your app thought reminds me of "Video Directly Observed Therapy" which is commonly used for things like tuberculosis medications (due to painful combo of lengthy duration of treatment, disproportionately affecting less well-off / well-educated people who are more challenged in compliance, and public health risk if inadequately treated). Several companies have also worked on pills with an component that detects when it's been ingested and beams a signal to a (external) device, thus "confirming" ingestion. Although of course, in a way, compliance is part and parcel of your drug's overall effectiveness. Not to dive into efficacy vs effectiveness and the larger considerations... but what if your drug "works" but nobody can be compliant? Of the top of my head, a few antibiotics are like, dosed 6 times a day, so I'd rather prescribe most patients a different one that's only taken twice a day, even if the 6 times a day antibiotic is technically "better" for their infection.
streptomycin 7y ago
If you like this post, check out the other posts on his blog too. It's fantastic content.
dennis_jeeves 7y ago
Because the supposed drug/therapy is so marginally effective that it need to be muddled up with tons of data (noise) and committees. Often I think it's just bureaucracy, and incompetence. Even if drug/therapy is massively effective, I'm sure the positive results would be lost in the bureaucracy.
neonate 7y ago
https://web.archive.org/web/20200502061205/https://blogs.sciencemag.org/pipeline/archives/2020/05/01/why-are-clinical-trials-so-complicated https://web.archive.org/web/20200502061205/https://blogs.sci...
gumby 7y ago
Having designed and run clinical trials and combo manufacturing (drug + device) in my past I can testify that this is right on. There's a huge effort (pushed by the FDA back on the manufacturer) to isolate variables and to ensure that whatever you want to say about your product you have actually verified to their satisfaction. As a person who also takes medication I'm glad about it. I also don't think people should harsh on the FDA because of their strictness. There's a lot of gaming attempted in dealing with them which makes them harder to work with, but in my experience they've been pretty decent to deal with. Of course we had a policy of no attempting to game the system.
- doctoring 7y ago
  On gaming: it's very interesting that, even within all of the byzantine rules and regulations and guidances how much gray area and leeway there can be. From a drug/device company perspective, which people in the FDA assigned as reviewers / contact points for you and how well you are able to articulate & convince them of your goals can play a huge role in whether you have to do X type of trial or adhere to Y rules. This is especially true at the boundaries (new types of testing, devices, technologies) that don't quite exactly fit the more well-trod paths. Not saying this in a judgmental way -- it's just the nature of dealing with extremely complex problems while also making sure people aren't harmed unnecessarily and that at the end when you say drug/device/tech A improves condition B we actually are sure it is true. And then there's the more malicious "gaming", fraud! Clinical research is obviously heavily incentivized toward positive trial outcomes, but for each site the act of just enrolling patients is incentivized, so you need to be damn sure the patient data (and the patients themselves!) in your study are real. And being sure is not easy or cheap. For fun, here's the FDA's list of people who have been blacklisted from performing clinical research: https://www.fda.gov/inspections-compliance-enforcement-and-criminal-investigations/compliance-actions-and-activities/fda-debarment-list-drug-product-applications https://www.fda.gov/inspections-compliance-enforcement-and-c... Their associated "Volume Page" lists (in excruciating bureaucratese) their "crimes", which may be unrelated illegal narcotics peddling (oops) or falsifying patient data.
  - gumby 7y ago
Gatsky 7y ago
Clinical trials are difficult. But when you consider they are a method to reliably generate new knowledge about complex systems (ie. humans) it doesn’t seem so bad. Actually they are remarkably efficient knowledge generators, compared to other areas of science, and have the advantage that a non-result is still useful up to a certain point.
anthony_doan 7y ago
Interned at the FDA. They take their stuff seriously. Clinical trials have 3 phases and it all have to be explained via statistic (design of experiment) and reproducible. They have standards for everything to enter the US drug market. Lots of biostatistics stuff too. Even when the drug enter the market, pharma companies stop testing the drugs. It is up to the FDA to keep track of side effects and toxicity that were not caught through the phases. (example: ambien sleeping drug) The people at FDA NCTR are wonderful, highly recommend interning there for a summer. They have some ML stuff. I've seen a project where they attempt to figure out what bug is in the food. One of the bug was midly toxic and they wanted a image classifier to figure it out via picture of a bug part. You can only have a certain ratio of bug parts in food.

106 points by yread 7y ago

For someone who never looked into the complexities of clinical trials, I thought that was a detailed and informative answer. It seems the key point to take away is, “The amount of noise in human clinical data can defy belief if you haven’t seen it in person”. :)

pfdietz 7y ago
A truth worth repeating!
zxexz 7y ago
I’ve been working in this space for a while now, and this still blows my mind. Patient-reported data is noisy, doctor’s notes infamously confusing and inconsistent in formatting. Even fields you can pull from the health system’s EHR directly will be in disparate formats across different hospitals, or even different departments. It would be easy to say it all should be standardized (and it should, for the most part), but workflow considerations keep standards from being adopted. What a data standard for oncology data would look like is very different from what one developed for ophthalmology. Not to mention UX - if you’re a doctor seeing 20 patients a day, an extra 15 minutes of data entry per patient could bring that number down to 15 patients or less. UX and data standards really does have a direct link to patient outcomes in healthcare!
- yread 7y ago
  Indeed, we had to recently work with 500k patients' appointments. Database table Appointment had a date and "actually met" flag - sounds simple enough, right? Nope - doctors were "meeting" dead people, different departments had different rules about whether phone calls were included and what did the "actually met" flag actually mean.

antipaul 7y ago

The saying in our industry is, "clinical trials are the most complicated thing in the world". EDIT: adding details: You have to mobilize large interdisciplinary teams to come up with products that are inherently very hard to invent, and which, after all that, can easily fail a trail and never see light of day, or can then still harm or kill people even after approved, all while operating in an understandbly extremely regulated and complicated environment. One example: One of the many piles of documents that you need to submit to a national health authority _just so they consider your product_ (before you can release to a single "user"!), is thorough documentation of how you _plan_ to manufacture the product, if it were approved. Just these text files can weigh _gigabytes_. Think how much effort it would take just to write something coherent that is so long! And of course, you have to get other teams to run a bunch of experiments to inform what to write. And this is even before you are actually producing the product for release. Oh, and maybe your product won't actually be approved, so all that goes to waste. And that's just the beginning. Because all of that excludes _any_ mention of the real deal, which is whether the product actually works or is safe. All of that requires its own experiments and documentation. And all that assumes that decades of prior research actually led to some useful prototype where you can start. And all that excludes any mention of ethical considerations like who pays for the research, what is a just price for a cancer drug - heck, whether you should even be trying to make a profit off of lifesaving drugs. If getting to a point like selling your app called Whats App for $18 billion (or whatever it was) would require a walk down the street, then getting a drug to market is like leaving the earth, flying past the furthest planets, and then trying to reach another galaxy.

Madmallard 7y ago
Why do I have a hard-time believing this? So many really bad drugs that are popularized and thrown with black-box warnings and such after the fact. I think the integrity of that system has been subverted.
- lobotryas 7y ago
  The key to taking a drug through trial is having enough cash for the trial and for your company to survive the time it takes to complete the trial successfully. Remember that said success is not guaranteed and you might have to write off years of work and start from scratch. This is why drugs are so expensive in the West.

Tomte 7y ago

> Once in a while, they’ll even say “Stop right now – your drug is so damn good that it’s unethical not to give it to everyone in the trial”, but years go by when that does not happen, anywhere in the industry. We've had such a case a short time ago (months?) in the news. Does anyone remember what that was about or even has a link?

alexrson 7y ago
You may be thinking of Tagrisso: https://www.statnews.com/2020/04/10/astrazeneca-tagrisso-study-ended-overwhelming-benefit/ https://www.statnews.com/2020/04/10/astrazeneca-tagrisso-stu...
antipaul 7y ago
Just because it's not reported in the Times, doesn't mean it doesn't happen. It actually can happen quite frequently. Trial protocols can pre-specify early stopping criteria: if we see drug A is better than drug B by a factor of X at time T, then we can declare superiority with given statistical confidence.
zxexz 7y ago
A kind of side-note, but for similar reasons, most interventional trials in the oncology space aren’t double-blinded for that reason. If you have cancer and sign up for a trial, you’ve likely run out of standard treatment options, so being given a placebo in a potentially life-saving therapy is not easy to justify. The different arms of the trial tend to vary in inclusion/exclusion criteria and doses, but only very rarely is there a placebo involved.
- tialaramex 7y ago
  Yup. I had Hodgkins many years ago. We've been treating Hodgkins for many decades with considerable success (~95% of young men with Hodgkins eventually die of something else and that may have improved further). And so it would be unethical to perform a placebo comparison because we're confident from anecdotal evidence (some patients refuse treatment) that it'll kill you. I was considered for (but ultimately did not qualify IIRC) a trial of a new Hodgkins treatment. If I had qualified and opted in all that would have meant is basically they toss a coin, heads I get the new treatment, tails I get the existing "Gold standard". As well as not using a placebo because it's a death sentence, they also can't truly blind the trial because the new regime is different drugs, with different side effects and a different calendar. The staff must know which they are giving you and even if they didn't just tell you it's not hard to look up online what the Gold Standard is and tell if you aren't getting it. So all they have left it's a two arm trial in which they've tried to match the arms up, and then compare the results. In my case the eventual trial results show the new regime was slightly worse, not better as hoped. Slightly more (statistically insignificant but not good news) deaths and worse side effects. Unfortunate, but there was only one way to find out.

choxi 7y ago

> The amount of noise in human clinical data can defy belief if you haven’t seen it in person I'm not surprised that patient compliance is a problem, but it still seems like an expensive burden to find patients physically near the clinical trials so they can come in to take the medication. It just seems like a logistical nightmare that software could help with. Could you build an app that requires patients to record themselves taking their medication under whatever conditions they're supposed to? You might have to manually audit all of the recordings, but the number of people that still fail to comply might be offset but how many more patients you can include in the study. There might be a lot of other things you could verify remotely using wearable tech, e.g. heart rate now that Apple's EKG monitoring is FDA approved. Great read, I'd like to see more "Chesterton's Fence" arguments for other highly regulated industries.

hpaavola 7y ago
"Could you build an app that requires patients to record themselves taking their medication under whatever conditions they're supposed to?" Yes. I work for a company (Signant Health) which does clinical trial platform as a service (among other things) and our platform has this feature that the trials can have steps which require the users to video certain things. And we are hiring https://signanthealth.com/careers/ https://signanthealth.com/careers/ :)
doctoring 7y ago
Many trials do NOT require patients to come in to take the medication, but there are of course some which are harder to take at home (whether intravenous or intrathecal or whatever). But compliance is certainly hard to assess and so important. Your app thought reminds me of "Video Directly Observed Therapy" which is commonly used for things like tuberculosis medications (due to painful combo of lengthy duration of treatment, disproportionately affecting less well-off / well-educated people who are more challenged in compliance, and public health risk if inadequately treated). Several companies have also worked on pills with an component that detects when it's been ingested and beams a signal to a (external) device, thus "confirming" ingestion. Although of course, in a way, compliance is part and parcel of your drug's overall effectiveness. Not to dive into efficacy vs effectiveness and the larger considerations... but what if your drug "works" but nobody can be compliant? Of the top of my head, a few antibiotics are like, dosed 6 times a day, so I'd rather prescribe most patients a different one that's only taken twice a day, even if the 6 times a day antibiotic is technically "better" for their infection.

streptomycin 7y ago

If you like this post, check out the other posts on his blog too. It's fantastic content.

dennis_jeeves 7y ago

Because the supposed drug/therapy is so marginally effective that it need to be muddled up with tons of data (noise) and committees. Often I think it's just bureaucracy, and incompetence. Even if drug/therapy is massively effective, I'm sure the positive results would be lost in the bureaucracy.

neonate 7y ago

https://web.archive.org/web/20200502061205/https://blogs.sciencemag.org/pipeline/archives/2020/05/01/why-are-clinical-trials-so-complicated https://web.archive.org/web/20200502061205/https://blogs.sci...

gumby 7y ago

Having designed and run clinical trials and combo manufacturing (drug + device) in my past I can testify that this is right on. There's a huge effort (pushed by the FDA back on the manufacturer) to isolate variables and to ensure that whatever you want to say about your product you have actually verified to their satisfaction. As a person who also takes medication I'm glad about it. I also don't think people should harsh on the FDA because of their strictness. There's a lot of gaming attempted in dealing with them which makes them harder to work with, but in my experience they've been pretty decent to deal with. Of course we had a policy of no attempting to game the system.

doctoring 7y ago
On gaming: it's very interesting that, even within all of the byzantine rules and regulations and guidances how much gray area and leeway there can be. From a drug/device company perspective, which people in the FDA assigned as reviewers / contact points for you and how well you are able to articulate & convince them of your goals can play a huge role in whether you have to do X type of trial or adhere to Y rules. This is especially true at the boundaries (new types of testing, devices, technologies) that don't quite exactly fit the more well-trod paths. Not saying this in a judgmental way -- it's just the nature of dealing with extremely complex problems while also making sure people aren't harmed unnecessarily and that at the end when you say drug/device/tech A improves condition B we actually are sure it is true. And then there's the more malicious "gaming", fraud! Clinical research is obviously heavily incentivized toward positive trial outcomes, but for each site the act of just enrolling patients is incentivized, so you need to be damn sure the patient data (and the patients themselves!) in your study are real. And being sure is not easy or cheap. For fun, here's the FDA's list of people who have been blacklisted from performing clinical research: https://www.fda.gov/inspections-compliance-enforcement-and-criminal-investigations/compliance-actions-and-activities/fda-debarment-list-drug-product-applications https://www.fda.gov/inspections-compliance-enforcement-and-c... Their associated "Volume Page" lists (in excruciating bureaucratese) their "crimes", which may be unrelated illegal narcotics peddling (oops) or falsifying patient data.
- gumby 7y ago

Gatsky 7y ago

Clinical trials are difficult. But when you consider they are a method to reliably generate new knowledge about complex systems (ie. humans) it doesn’t seem so bad. Actually they are remarkably efficient knowledge generators, compared to other areas of science, and have the advantage that a non-result is still useful up to a certain point.

anthony_doan 7y ago

Interned at the FDA. They take their stuff seriously. Clinical trials have 3 phases and it all have to be explained via statistic (design of experiment) and reproducible. They have standards for everything to enter the US drug market. Lots of biostatistics stuff too. Even when the drug enter the market, pharma companies stop testing the drugs. It is up to the FDA to keep track of side effects and toxicity that were not caught through the phases. (example: ambien sleeping drug) The people at FDA NCTR are wonderful, highly recommend interning there for a summer. They have some ML stuff. I've seen a project where they attempt to figure out what bug is in the food. One of the bug was midly toxic and they wanted a image classifier to figure it out via picture of a bug part. You can only have a certain ratio of bug parts in food.