7 ms·

It’s getting harder to do a randomized trial for some drugs

94 points by gautamcgoel 4y ago

dang 4y ago
I replaced the baity title with a sentence from the article that seems to say what it's about. If someone can find a more representative sentence, we can change it again. (From https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html: "Please use the original title, unless it is misleading or linkbait")
- vxNsr 4y ago
  Thanks in this case it’s not only less baity, but also more informative
mjfl 4y ago
This article simultaneously demands that we need to NOT abandon the old system of large scale clinical trials, and not go towards faster means of approving drugs, while at the same time lamenting that CAR-T remains unapproved for certain cancers large scale trials have shown it to not be sufficiently effective. It says that we need to "make trials better" while not proposing any means to do that. What a waste of time.
- balsam 4y ago
  As Derek Lowe summarized in the accompanying blogpost[0], the main recommendations of this article are: “better decisions about what data to collect and better ways to collect it.” Yeah, sometimes people dont get their point across very well, but it is there. [0]https://www.science.org/content/blog-post/better-trials-someday https://www.science.org/content/blog-post/better-trials-some...
  - taeric 4y ago
    I was hoping to find a bit more than, "do better." What are the decisions? How can they be made better? What data? Why aren't we collecting it already?
    - balsam 4y ago
      In TFA they mention a insightful case study: “Most clinical trials try to track every single possible data point about patients, thus inflating costs. But Landry and Horby’s trial could carefully strip down the data needed to the essentials, focusing on information that could easily be collected from medical records. “ A couple sentences later: “The result was a single study, RECOVERY, that determined more than any other what medicines could help Covid patients.” As for why we arent’t doing better, just last week there was a HN post that I personally enjoyed: https://news.ycombinator.com/item?id=33545209 https://news.ycombinator.com/item?id=33545209 EDIT: The RECOVERY trial has its own wikipedia page, more references therein if you are interested in the details https://en.m.wikipedia.org/wiki/RECOVERY_Trial https://en.m.wikipedia.org/wiki/RECOVERY_Trial
      - taeric 4y ago
        Yeah... This don't really sound prescriptive. The "single study" that looked at COVID is particularly annoying. It wasn't done in isolation with no knowledge of every other study. Such that there is virtually no way to jump straight to that study. Specifically for stripping down the data. That was informed by every other study.
    - vxNsr 4y ago
      Yup “do better” is the generic siren of those who like to complain but never have anything to contribute.
  - ekianjo 4y ago
    better means what?
    - rocqua 4y ago
      Beter decisions about what to collect would, for example be A higher ratio of 'progress towards definitive answer' per dollar spend. Perhaps with a smidge of, making people more likely to participate. Better ways to collect data has two routes for improvement. The boring one is automation. Just chuck smart watches on the subjects and have those track data on a hourly basis rather than monthly doctor visits. The exciting but scary one is standardized health records keeping that allows tracking outcomes. That also requires a process for analyzing that data on a national scale whilst preserving privacy. The crypto to do that exists, but getting trust will be difficult. There could be other ways of doing analysis without hurting privacy too much. This system could then be used whenever 'real-world-data' is appropriate. Importantly, it could also be used to track data from the randomized medical trials!
meltyness 4y ago
Medicine has problems that aren't even spelled right theoretically. EHR portability is just a symptom of that issue. Clever solutions like MYCIN, a decision tree mapping symptoms and conditions to possible diagnoses, are largely ignored because it is more profitable to maintain a facade and saddle increasingly siloed clinicians with debt.
- Animats 4y ago
  MYCIN, the rule-based system from the 1980s? It was once touted as "AI", but, like most knowledge based systems, it just takes a set of hand-coded rules and interprets them in a rather simple way.
  - SoftTalker 4y ago
    Most of what you learn in medical school is a set of hand-coded rules. It has a fancier legal name, "standard of care" but that's pretty much what it is.
    - Retric 4y ago
      Only for routine cases. Medicine is full of one of uncertainty, edge cases, and new research. The difficulty is identifying what’s novel vs routine, think of the first few COVID cases and the 2022 monkey pox outbreak. When exactly do you sound the alarm?
      - zarzavat 4y ago
        That’s completely true but doctors also miss things, or make errors, that would have been trivially caught by a deterministic rule-based system. Just look at how successful checklists have been in surgery (or air transport), that’s a very simple rule-based system that has saved countless lives. Helping doctors to diagnose effectively requires someone to ask them the right questions. All too often the job of asking questions is left to the patient.
        4y ago
n8ta 4y ago
Great read! So why are trials getting more expensive? The crux of the article was triald aren't being done because costs have risen, but why have costs risen? Have they risen faster than drug prices?
- joe_the_user 4y ago
  Very quick searching seems to indicate increased complexity of trials is a big factor. And trials are more complex because most of the drugs being tested are incremental improvements on existing drugs. “Drugs are more expensive to test when they have smaller effects that require observing more patients for longer periods of time,” [1] Because you have to observe more people over a longer period of time. That also tempers the article's enthusiasm for biological technologies. The problem with health technology is that every single step is uncertain, every single step needs to be tested. It's "artisanal" to an extreme extent. There is no equivalent to Moore's law. Each step costs more, not less. Well, until we reach the point we can grow who organs easily. Then thing might change. [1] Title: "From concept to clearance: Why it’s so expensive to develop new CV [cardiovascular] drugs" https://cardiovascularbusiness.com/topics/healthcare-management/healthcare-economics/why-its-so-expensive-develop-new-cv-drugs https://cardiovascularbusiness.com/topics/healthcare-managem...
- rocqua 4y ago
  There are hints of tracking too many data points, and perhaps too much manual tracking. Also the story that, for very profitable drugs the approach is about speed of trials rather than cost. If that is pervasive then it might become standard. Heck, if that is where the money is, that is where carreers will gravitate to.
- DigDugDude 4y ago
  The majority of cost comes from associated procedures and admin/staff costs (not including that of physicians and nurses).
balsam 4y ago
Derek Lowe (the Matthew Levine of biotech) blogged about this article and topic recently: https://www.science.org/content/blog-post/better-trials-someday https://www.science.org/content/blog-post/better-trials-some...
foreigner 4y ago
TLDR: For-profit healthcare systems optimise for profit, not health.
- mindslight 4y ago
  Would you also say that for-profit food production systems optimize for profit, not food? I'd personally say that anyone going hungry in the modern age is a travesty. But to remedy that, I'd leverage existing for-profit food production rather than thinking to overhaul the whole system with less profit motive. The incentives of the healthcare system are deeply screwed up though. Much of this due to existing regulations, with the rest being momentum and emergent behavior due to those broken incentives. But just to be clear - I'm not arguing in favor of deregulation, but rather sweeping reregulation. For example, there should be no such thing as an "HMO".
refurb 4y ago
Retrospective real world data and mining from EHR is useful, but it’s not as robust as a prospective randomized clinical trial. There is no protocol so not all data is captured. You’re also relying on the robustness of the data capture which is no where close to as good as a clinical trial because it’s just clinical practice. As such, you need to filter and refine data sets, which opens up the green jelly bean problem. Filters become subjective based on “best thinking”. Which is fine, but guess who looks at the holes in the data and decides they won’t pay for it? Public and private healthcare payers. I’ve been involved in those negotiations. Non-RCT data is “suspect” and it’s not unusual to hear “go back and conduct another trial so you have reliable data”. They are great complements, but not a substitute for RCTs.
- rocqua 4y ago
  This matches what the author says no? He argues that clinical trials (i.e. deciding whether to approve a drug) should always be randomized. At the same time he believes there is lots of potential for real-world-data in other steps. Pre-trial simulation, post approval outcome tracking, automated drug-interaction analysis, etc. The data analysis is incredibly valuable, even if not suited for drug approval decisions.
thedrbrian 4y ago
I dunno why they bother with trials. Just get the media to whip up a frenzy about that ever the drug is supposed to treat and slap an EUA on it. Job done. Line goes up

94 points by gautamcgoel 4y ago

dang 4y ago

I replaced the baity title with a sentence from the article that seems to say what it's about. If someone can find a more representative sentence, we can change it again. (From https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html: "Please use the original title, unless it is misleading or linkbait")

vxNsr 4y ago
Thanks in this case it’s not only less baity, but also more informative

mjfl 4y ago

This article simultaneously demands that we need to NOT abandon the old system of large scale clinical trials, and not go towards faster means of approving drugs, while at the same time lamenting that CAR-T remains unapproved for certain cancers large scale trials have shown it to not be sufficiently effective. It says that we need to "make trials better" while not proposing any means to do that. What a waste of time.

balsam 4y ago
As Derek Lowe summarized in the accompanying blogpost[0], the main recommendations of this article are: “better decisions about what data to collect and better ways to collect it.” Yeah, sometimes people dont get their point across very well, but it is there. [0]https://www.science.org/content/blog-post/better-trials-someday https://www.science.org/content/blog-post/better-trials-some...
- taeric 4y ago
  I was hoping to find a bit more than, "do better." What are the decisions? How can they be made better? What data? Why aren't we collecting it already?
  - balsam 4y ago
    In TFA they mention a insightful case study: “Most clinical trials try to track every single possible data point about patients, thus inflating costs. But Landry and Horby’s trial could carefully strip down the data needed to the essentials, focusing on information that could easily be collected from medical records. “ A couple sentences later: “The result was a single study, RECOVERY, that determined more than any other what medicines could help Covid patients.” As for why we arent’t doing better, just last week there was a HN post that I personally enjoyed: https://news.ycombinator.com/item?id=33545209 https://news.ycombinator.com/item?id=33545209 EDIT: The RECOVERY trial has its own wikipedia page, more references therein if you are interested in the details https://en.m.wikipedia.org/wiki/RECOVERY_Trial https://en.m.wikipedia.org/wiki/RECOVERY_Trial
    - taeric 4y ago
      Yeah... This don't really sound prescriptive. The "single study" that looked at COVID is particularly annoying. It wasn't done in isolation with no knowledge of every other study. Such that there is virtually no way to jump straight to that study. Specifically for stripping down the data. That was informed by every other study.
  - vxNsr 4y ago
    Yup “do better” is the generic siren of those who like to complain but never have anything to contribute.
- ekianjo 4y ago
  better means what?
  - rocqua 4y ago
    Beter decisions about what to collect would, for example be A higher ratio of 'progress towards definitive answer' per dollar spend. Perhaps with a smidge of, making people more likely to participate. Better ways to collect data has two routes for improvement. The boring one is automation. Just chuck smart watches on the subjects and have those track data on a hourly basis rather than monthly doctor visits. The exciting but scary one is standardized health records keeping that allows tracking outcomes. That also requires a process for analyzing that data on a national scale whilst preserving privacy. The crypto to do that exists, but getting trust will be difficult. There could be other ways of doing analysis without hurting privacy too much. This system could then be used whenever 'real-world-data' is appropriate. Importantly, it could also be used to track data from the randomized medical trials!

meltyness 4y ago

Medicine has problems that aren't even spelled right theoretically. EHR portability is just a symptom of that issue. Clever solutions like MYCIN, a decision tree mapping symptoms and conditions to possible diagnoses, are largely ignored because it is more profitable to maintain a facade and saddle increasingly siloed clinicians with debt.

Animats 4y ago
MYCIN, the rule-based system from the 1980s? It was once touted as "AI", but, like most knowledge based systems, it just takes a set of hand-coded rules and interprets them in a rather simple way.
- SoftTalker 4y ago
  Most of what you learn in medical school is a set of hand-coded rules. It has a fancier legal name, "standard of care" but that's pretty much what it is.
  - Retric 4y ago
    Only for routine cases. Medicine is full of one of uncertainty, edge cases, and new research. The difficulty is identifying what’s novel vs routine, think of the first few COVID cases and the 2022 monkey pox outbreak. When exactly do you sound the alarm?
    - zarzavat 4y ago
      That’s completely true but doctors also miss things, or make errors, that would have been trivially caught by a deterministic rule-based system. Just look at how successful checklists have been in surgery (or air transport), that’s a very simple rule-based system that has saved countless lives. Helping doctors to diagnose effectively requires someone to ask them the right questions. All too often the job of asking questions is left to the patient.
      - 4y ago

n8ta 4y ago

Great read! So why are trials getting more expensive? The crux of the article was triald aren't being done because costs have risen, but why have costs risen? Have they risen faster than drug prices?

joe_the_user 4y ago
Very quick searching seems to indicate increased complexity of trials is a big factor. And trials are more complex because most of the drugs being tested are incremental improvements on existing drugs. “Drugs are more expensive to test when they have smaller effects that require observing more patients for longer periods of time,” [1] Because you have to observe more people over a longer period of time. That also tempers the article's enthusiasm for biological technologies. The problem with health technology is that every single step is uncertain, every single step needs to be tested. It's "artisanal" to an extreme extent. There is no equivalent to Moore's law. Each step costs more, not less. Well, until we reach the point we can grow who organs easily. Then thing might change. [1] Title: "From concept to clearance: Why it’s so expensive to develop new CV [cardiovascular] drugs" https://cardiovascularbusiness.com/topics/healthcare-management/healthcare-economics/why-its-so-expensive-develop-new-cv-drugs https://cardiovascularbusiness.com/topics/healthcare-managem...
rocqua 4y ago
There are hints of tracking too many data points, and perhaps too much manual tracking. Also the story that, for very profitable drugs the approach is about speed of trials rather than cost. If that is pervasive then it might become standard. Heck, if that is where the money is, that is where carreers will gravitate to.
DigDugDude 4y ago
The majority of cost comes from associated procedures and admin/staff costs (not including that of physicians and nurses).

balsam 4y ago

Derek Lowe (the Matthew Levine of biotech) blogged about this article and topic recently: https://www.science.org/content/blog-post/better-trials-someday https://www.science.org/content/blog-post/better-trials-some...

foreigner 4y ago

TLDR: For-profit healthcare systems optimise for profit, not health.

mindslight 4y ago
Would you also say that for-profit food production systems optimize for profit, not food? I'd personally say that anyone going hungry in the modern age is a travesty. But to remedy that, I'd leverage existing for-profit food production rather than thinking to overhaul the whole system with less profit motive. The incentives of the healthcare system are deeply screwed up though. Much of this due to existing regulations, with the rest being momentum and emergent behavior due to those broken incentives. But just to be clear - I'm not arguing in favor of deregulation, but rather sweeping reregulation. For example, there should be no such thing as an "HMO".

refurb 4y ago

Retrospective real world data and mining from EHR is useful, but it’s not as robust as a prospective randomized clinical trial. There is no protocol so not all data is captured. You’re also relying on the robustness of the data capture which is no where close to as good as a clinical trial because it’s just clinical practice. As such, you need to filter and refine data sets, which opens up the green jelly bean problem. Filters become subjective based on “best thinking”. Which is fine, but guess who looks at the holes in the data and decides they won’t pay for it? Public and private healthcare payers. I’ve been involved in those negotiations. Non-RCT data is “suspect” and it’s not unusual to hear “go back and conduct another trial so you have reliable data”. They are great complements, but not a substitute for RCTs.

rocqua 4y ago
This matches what the author says no? He argues that clinical trials (i.e. deciding whether to approve a drug) should always be randomized. At the same time he believes there is lots of potential for real-world-data in other steps. Pre-trial simulation, post approval outcome tracking, automated drug-interaction analysis, etc. The data analysis is incredibly valuable, even if not suited for drug approval decisions.

thedrbrian 4y ago

I dunno why they bother with trials. Just get the media to whip up a frenzy about that ever the drug is supposed to treat and slap an EUA on it. Job done. Line goes up