4 ms·

That is correct, the people that have already died are unable to act as effective controls. To establish the medicine causes the disease to go away, you have to

That is correct, the people that have already died are unable to act as effective controls. To establish the medicine causes the disease to go away, you have to use something like a randomized trial. Otherwise, causation is not established. It’s a very difficult problem and a lot of smart people have try to come up with better systems.

mitthrowaway2 3y ago
With all due respect, why? This seems fallaciously wrong. It is rather akin to saying that I can't tell whether seatbelts reduce car-crash fatalities simply by comparing collision fatalities from 1967 seatbelt-lacking car models with otherwise-identical 1968 seatbelt-equipped models; I would need to actually randomly and secretly sabotage a certain percentage of the seatbelts from 1968-model cars, while deliberately ignoring everything I know about prior fatality statistics. Even if every automobile collision without seatbelts was fatal, and every collision after seatbelts became mandatory was survived, even if nothing about the cars or driving conditions had changed, you're implying that it wouldn't provide any evidence of the efficacy of seatbelts, unless we conducted a new trial from scratch with a randomized control group as though we had no prior data? What is the actual difference, in terms of causal evidence, between someone who died of the same disease before the study officially begins, and someone in the "control group" who dies after the study begins?
- shadowgovt 3y ago
  Almost nothing about medicine operates nearly as cleanly as that example; that's one of the major challenges with medical science. The seatbelt analogy breaks down in this context because the hypothetical of "Even if every automobile collision without seatbelts was fatal, and every collision after seatbelts became mandatory was survived, even if nothing about the cars or driving conditions had changed" doesn't reflect medical reality (and in the rare instances where it does, i.e. when improvement for the test group becomes obvious, immediate, and apparently side-effect free, it can become morally imperative to discontinue the control group). In medicine, you get confounding factors like change in standard of care year-to-year and facility-to-facility. You almost never get the kind of "clean-room" trials you get in mechanical engineering (and, indeed, the processes to create them would often themselves be unethical). So the best you usually get is trying to reason around (to torture the analogy a bit) "Well we introduced seatbelts three years ago and relative to the year prior to that, fatalities went way down... Oh wait, but COVID also happened the same year we introduced seatbelts and everyone stopped driving" if you compare a current trial to past non-trial standard care as the control group.
  - mitthrowaway2 3y ago
    Isn't this just the standard challenge of separation of signal and noise that exists in all disciplines, though? It seems that it would be better dealt with by general Bayesian methods than by strict adherence to a controlled-trial process that requires ignoring or even discarding large quantities of data (including every patient who ever had the disease prior to the start of the trial, which for rare diseases, might even be the majority of the data!)
    - shadowgovt 3y ago
      > It seems that it would be better dealt with by general Bayesian methods than by strict adherence to a controlled-trial process that requires ignoring or even discarding large quantities of data What is the Bayesian prior on "After receiving the COVID vaccination, patient died when they were struck by lightning?" This is what I mean when I say the 'clean room' practices that are standard for most other fields of science would be considered unethical for medical trials. You can't, generally, lock people in a box for months to eliminate other confounding factors when trialing medicine; the best approach we have, therefore, is to minimize confounding factors by minimizing variables in space, time, &c. Yes, if we could lab-rat humans, keep them in controlled environments perpetually, we could use the previous in-a-box state as a comparison to the current state, probably. That's obviously a non-starter. (There are some categories of medical research where time-variance has been used; it's just not generally an acceptable signal source, IIUC, for drug and therapy trials. If for no other reason than not nearly enough information has been captured about unrelated past patients to be compared to a current trial because relevant information to monitor was not known when monitoring past patients).
      - mitthrowaway2 3y ago
        > What is the Bayesian prior on "After receiving the COVID vaccination, patient died when they were struck by lightning?" Mathematically not difficult to deal with, which is why you intuitively know the answer already, because it's approximately the same way your brain works naturally. The chance of dying when you are struck by lightning is about 10%,[1] which completely overwhelms the chance of the vaccine coincidentally killing you at the same moment by several orders of magnitude; while you could use the this datapoint if you want, Bayesian analysis will still confirm that it has approximately zero evidence supporting (or refuting) that the vaccine causes death, so Bayes' theorem effectively drops that patient out of the analysis. Which is the same thing you'd probably have done intuitively, unless some kind of formal controlled-test study rule prevented you from doing what would normally be the correct procedure. (Or worse, forced you to subject an unvaccinated control-group patient to a lightning strike, just to maintain control...?) But, since your patient died from the lightning strike rather than surviving it, there is a small signal present that points towards "the vaccine increased the chance of the lightning strike being fatal", which is a postulate that deserves a high initial skepticism until someone can propose a mechanism. But if you notice that multiple people die from lightning strikes after getting vaccinated, such that their lightning-strike survival rate is below the unvaccinated population's, you can start to update on that, and if it keeps happening, it becomes worth your time to investigate a potential mechanistic explanation. Like, the vaccination causing people to be less sweaty somehow. [1] https://www.britannica.com/one-good-fact/what-are-the-odds-of-surviving-a-lighting-strike https://www.britannica.com/one-good-fact/what-are-the-odds-o...
- youainti 3y ago
  The process in which a treatment is administered might change the way people behave, changing outcomes. Trials with control groups are easier to justify as having avoided these issues. If you're interested in learning more, I would recommend learning about Judea Pearl's do-calculus, specifically the backdoor criterion. Although it isn't the end-all of causal analysis, it is a very useful non-experimental approach that will help you develop an appreciation for the benefits of controlled experiments. As far as seat belts go, that's actually been something discussed a bit in economics. The main concern is if adding seat belts changes how people drive (confounding the actual effects of seat belts). The solution was to measure the protective effect of seat belts on occupants of the passenger seat in collisions involving drunk drivers.
nick__m 3y ago
If a disease at a certain well defined stage has a 100% fatality rate after 6 months, why would you need a double blind trial if some patients are still alive 1 year after taking your drug ? The causality is pretty clear and while you might not claim that it double the life expectancy it would IMHO be immoral not to fast track the approval of that drug.
Zak 3y ago
People who already died are less effective controls than double-blind controls getting a placebo, but they're not ineffective. Given a treatment with a large effect size like penicillin, a causal link will be very easy to establish even with a suboptimal control group.

mitthrowaway2 3y ago

With all due respect, why? This seems fallaciously wrong. It is rather akin to saying that I can't tell whether seatbelts reduce car-crash fatalities simply by comparing collision fatalities from 1967 seatbelt-lacking car models with otherwise-identical 1968 seatbelt-equipped models; I would need to actually randomly and secretly sabotage a certain percentage of the seatbelts from 1968-model cars, while deliberately ignoring everything I know about prior fatality statistics. Even if every automobile collision without seatbelts was fatal, and every collision after seatbelts became mandatory was survived, even if nothing about the cars or driving conditions had changed, you're implying that it wouldn't provide any evidence of the efficacy of seatbelts, unless we conducted a new trial from scratch with a randomized control group as though we had no prior data? What is the actual difference, in terms of causal evidence, between someone who died of the same disease before the study officially begins, and someone in the "control group" who dies after the study begins?

shadowgovt 3y ago
Almost nothing about medicine operates nearly as cleanly as that example; that's one of the major challenges with medical science. The seatbelt analogy breaks down in this context because the hypothetical of "Even if every automobile collision without seatbelts was fatal, and every collision after seatbelts became mandatory was survived, even if nothing about the cars or driving conditions had changed" doesn't reflect medical reality (and in the rare instances where it does, i.e. when improvement for the test group becomes obvious, immediate, and apparently side-effect free, it can become morally imperative to discontinue the control group). In medicine, you get confounding factors like change in standard of care year-to-year and facility-to-facility. You almost never get the kind of "clean-room" trials you get in mechanical engineering (and, indeed, the processes to create them would often themselves be unethical). So the best you usually get is trying to reason around (to torture the analogy a bit) "Well we introduced seatbelts three years ago and relative to the year prior to that, fatalities went way down... Oh wait, but COVID also happened the same year we introduced seatbelts and everyone stopped driving" if you compare a current trial to past non-trial standard care as the control group.
- mitthrowaway2 3y ago
  Isn't this just the standard challenge of separation of signal and noise that exists in all disciplines, though? It seems that it would be better dealt with by general Bayesian methods than by strict adherence to a controlled-trial process that requires ignoring or even discarding large quantities of data (including every patient who ever had the disease prior to the start of the trial, which for rare diseases, might even be the majority of the data!)
  - shadowgovt 3y ago
    > It seems that it would be better dealt with by general Bayesian methods than by strict adherence to a controlled-trial process that requires ignoring or even discarding large quantities of data What is the Bayesian prior on "After receiving the COVID vaccination, patient died when they were struck by lightning?" This is what I mean when I say the 'clean room' practices that are standard for most other fields of science would be considered unethical for medical trials. You can't, generally, lock people in a box for months to eliminate other confounding factors when trialing medicine; the best approach we have, therefore, is to minimize confounding factors by minimizing variables in space, time, &c. Yes, if we could lab-rat humans, keep them in controlled environments perpetually, we could use the previous in-a-box state as a comparison to the current state, probably. That's obviously a non-starter. (There are some categories of medical research where time-variance has been used; it's just not generally an acceptable signal source, IIUC, for drug and therapy trials. If for no other reason than not nearly enough information has been captured about unrelated past patients to be compared to a current trial because relevant information to monitor was not known when monitoring past patients).
    - mitthrowaway2 3y ago
      > What is the Bayesian prior on "After receiving the COVID vaccination, patient died when they were struck by lightning?" Mathematically not difficult to deal with, which is why you intuitively know the answer already, because it's approximately the same way your brain works naturally. The chance of dying when you are struck by lightning is about 10%,[1] which completely overwhelms the chance of the vaccine coincidentally killing you at the same moment by several orders of magnitude; while you could use the this datapoint if you want, Bayesian analysis will still confirm that it has approximately zero evidence supporting (or refuting) that the vaccine causes death, so Bayes' theorem effectively drops that patient out of the analysis. Which is the same thing you'd probably have done intuitively, unless some kind of formal controlled-test study rule prevented you from doing what would normally be the correct procedure. (Or worse, forced you to subject an unvaccinated control-group patient to a lightning strike, just to maintain control...?) But, since your patient died from the lightning strike rather than surviving it, there is a small signal present that points towards "the vaccine increased the chance of the lightning strike being fatal", which is a postulate that deserves a high initial skepticism until someone can propose a mechanism. But if you notice that multiple people die from lightning strikes after getting vaccinated, such that their lightning-strike survival rate is below the unvaccinated population's, you can start to update on that, and if it keeps happening, it becomes worth your time to investigate a potential mechanistic explanation. Like, the vaccination causing people to be less sweaty somehow. [1] https://www.britannica.com/one-good-fact/what-are-the-odds-of-surviving-a-lighting-strike https://www.britannica.com/one-good-fact/what-are-the-odds-o...
youainti 3y ago
The process in which a treatment is administered might change the way people behave, changing outcomes. Trials with control groups are easier to justify as having avoided these issues. If you're interested in learning more, I would recommend learning about Judea Pearl's do-calculus, specifically the backdoor criterion. Although it isn't the end-all of causal analysis, it is a very useful non-experimental approach that will help you develop an appreciation for the benefits of controlled experiments. As far as seat belts go, that's actually been something discussed a bit in economics. The main concern is if adding seat belts changes how people drive (confounding the actual effects of seat belts). The solution was to measure the protective effect of seat belts on occupants of the passenger seat in collisions involving drunk drivers.

nick__m 3y ago

If a disease at a certain well defined stage has a 100% fatality rate after 6 months, why would you need a double blind trial if some patients are still alive 1 year after taking your drug ? The causality is pretty clear and while you might not claim that it double the life expectancy it would IMHO be immoral not to fast track the approval of that drug.

Zak 3y ago

People who already died are less effective controls than double-blind controls getting a placebo, but they're not ineffective. Given a treatment with a large effect size like penicillin, a causal link will be very easy to establish even with a suboptimal control group.