4 ms·

This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in

by taeric 3mo ago

This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case. Should we demand an RCT before we accept evidence? Of course not. At some point you do have to make a choice on things. And it should be noted that most drugs do have early cutoff criteria if the evidence is strong enough that it is working. It isn't like people are wanting to withhold good treatments from the world. Adding controls and randomizing them, though, has proven to be highly effective at helping progress.

A_D_E_P_T 3mo ago
> "This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case." If you have enough data, you can smooth out individual fluctuations due to things like drug interactions, non-compliance, etc. (And indeed you might discover drug interactions!) Observational trials ultimately mirror how drugs are used in the real world. > "Adding controls and randomizing them, though, has proven to be highly effective at helping progress." I would argue just the opposite. Demands for increasingly byzantine trials have ballooned the costs associated with drug development, and have slowed things to a crawl. There's a reason the field's golden age was in the 1940s and 1950s, and it's not just "low hanging fruit." Today nobody in their right mind wants to work in drug development when they could work in tech or even finance.
- taeric 3mo ago
  Right, but you are just relying on a different form of random, there. The whole point of making controls and then building experiments on changing them, is to get more power from fewer observations. No? Again, it is off to think that one is automatically superior to the other. Certainly to the exclusion of the other. And that is what feels off with the framing of the parent post. I am perfectly fine saying you should use both observational and controlled trials. But I think it is also wrong to think you don't have to build experiments to test interventions. This is why you put metrics in your service code. So that you can observe them behave and look for things to change. This is also why you do test cases on your code, so that you can specifically target your change. Now, I fully back the idea that just A/B testing something doesn't automatically mean you learn something true. But neither does observing a strong outcome on uncontrolled data.
  - estearum 3mo ago
    Yeah, GP is basically saying: "Large controlled experiments are costly and can hurt people who opt-in to informed consent. Instead, we should do significantly, significantly larger experiments, with undefined success/failure conditions, and no informed consent." Insane opinion
- RandomLensman 3mo ago
  How do you get enough data? If, for example, you need a lot of people in the sample, that might not be so easy. In the abstract, should it not come done to what is the best experimental design for each case?
- deleted 3mo ago
  [deleted]
- __alexs 3mo ago
  You are simultaneously arguing for a more complex and nuanced testing approach that demands much higher quantities of data as a result, and also against RCTs, which perhaps rightly you've identified as having suffered from the same kind of cost disease as all other health care in the USA. I can't help but feel like you've identified the wrong root cause here.
- tremon 3mo ago
  > There's a reason the field's golden age was in the 1940s and 1950s Yes, that was because of things like https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment https://en.wikipedia.org/wiki/Stateville_Penitentiary_Malaria_Study https://en.wikipedia.org/wiki/Stateville_Penitentiary_Malari... https://en.wikipedia.org/wiki/Operation_Sea-Spray https://en.wikipedia.org/wiki/Operation_Sea-Spray https://en.wikipedia.org/wiki/Nazi_human_experimentation https://en.wikipedia.org/wiki/Nazi_human_experimentation I understand that certain people are salivating at the thought of a return to those times; I'm not one of them.
  - smalltorch 3mo ago
    >Operation Sea-Spray was a 1950 U.S. Navy secret biological warfare experiment in which Serratia marcescens and Bacillus globigii bacteria were sprayed over the San Francisco Bay Area in California in order to determine how vulnerable a city like San Francisco would be to a bioweapon attack. There has been speculation that the experiment may have contributed to one death and at least ten illnesses. Wow.
  - estearum 3mo ago
    Also, despite GP's dismissal of the "low-hanging fruit" hypothesis, it is obviously true that we've found the easy-to-discover drugs, and therefore drugs would get harder and harder to find. A flippant dismissal in an HN comment does not actually negate reality.
- estearum 3mo ago
  > Demands for increasingly byzantine trials This is silly. FDA has essentially one requirement: prove that your drug is safe and effective. The reason trial designs get more and more byzantine is because the drugs themselves work less-and-less well. They're far more nuanced and precise. The experiments have to be extremely well-controlled, and then this has to balance against cost/timeline of the trial, and that's why sponsors choose to use byzantine trial designs.
  - A_D_E_P_T 3mo ago
    Proving "efficacy" -- which is the difficult and expensive part -- should not be necessary, and the government increasingly moves its own goalposts as to what the word efficacy even means. Simple as that. Postmarketing surveillance can easily determine what's effective and what's not, and medical orgs can adjust.
    - estearum 3mo ago
      Brilliant idea. This way both actual working drugs and vitamin-aisle potions look identical to consumers. Neither (or both?) can make claims to their effectiveness since they're both backed by the same (lack of) actual knowledge. This is especially great because it puts anyone who actually wants to actually make a working drug at a significant disadvantage. It'll take them longer to get to market, cost them a billion dollars more, then their medicine gets to sit next to a thousand variations of "Vitamin C for Leukemia" that all cost a lot less. There would be virtually no incentive for anyone to make an actual drug. > Postmarketing surveillance can easily determine what's effective and what's not, and medical orgs can adjust. "Easily" is doing a ton of work. Postmarketing surveillance can sometimes give low-confidence signal as to what's effective and what's not.
      - qsera 3mo ago
        >There would be virtually no incentive for anyone to make an actual drug. Dumb question: Wouldn't the market discover what works and what does not automatically, like any other product?
- graemep 3mo ago
  > Demands for increasingly byzantine trials have ballooned the costs associated with drug development, Even if that is true, is it an intrinsic problem with trials or just bad regulation? If it is the latter then you need to change the regulations? Is the problem global - is every regulator everywhere demanding byzantine trails?
  - estearum 3mo ago
    Regulators don't demand byzantine trials. They'd prefer simpler ones for a million different reasons. Byzantine trial designs exist because it's incredibly hard to prove your drug works even in an RCT. But GP thinks you can just look at EMR data and ta-da now you know, lol.

by taeric 3mo ago

A_D_E_P_T 3mo ago

> "This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case." If you have enough data, you can smooth out individual fluctuations due to things like drug interactions, non-compliance, etc. (And indeed you might discover drug interactions!) Observational trials ultimately mirror how drugs are used in the real world. > "Adding controls and randomizing them, though, has proven to be highly effective at helping progress." I would argue just the opposite. Demands for increasingly byzantine trials have ballooned the costs associated with drug development, and have slowed things to a crawl. There's a reason the field's golden age was in the 1940s and 1950s, and it's not just "low hanging fruit." Today nobody in their right mind wants to work in drug development when they could work in tech or even finance.

taeric 3mo ago
Right, but you are just relying on a different form of random, there. The whole point of making controls and then building experiments on changing them, is to get more power from fewer observations. No? Again, it is off to think that one is automatically superior to the other. Certainly to the exclusion of the other. And that is what feels off with the framing of the parent post. I am perfectly fine saying you should use both observational and controlled trials. But I think it is also wrong to think you don't have to build experiments to test interventions. This is why you put metrics in your service code. So that you can observe them behave and look for things to change. This is also why you do test cases on your code, so that you can specifically target your change. Now, I fully back the idea that just A/B testing something doesn't automatically mean you learn something true. But neither does observing a strong outcome on uncontrolled data.
- estearum 3mo ago
  Yeah, GP is basically saying: "Large controlled experiments are costly and can hurt people who opt-in to informed consent. Instead, we should do significantly, significantly larger experiments, with undefined success/failure conditions, and no informed consent." Insane opinion
RandomLensman 3mo ago
How do you get enough data? If, for example, you need a lot of people in the sample, that might not be so easy. In the abstract, should it not come done to what is the best experimental design for each case?
deleted 3mo ago
[deleted]
__alexs 3mo ago
You are simultaneously arguing for a more complex and nuanced testing approach that demands much higher quantities of data as a result, and also against RCTs, which perhaps rightly you've identified as having suffered from the same kind of cost disease as all other health care in the USA. I can't help but feel like you've identified the wrong root cause here.
tremon 3mo ago
> There's a reason the field's golden age was in the 1940s and 1950s Yes, that was because of things like https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment https://en.wikipedia.org/wiki/Stateville_Penitentiary_Malaria_Study https://en.wikipedia.org/wiki/Stateville_Penitentiary_Malari... https://en.wikipedia.org/wiki/Operation_Sea-Spray https://en.wikipedia.org/wiki/Operation_Sea-Spray https://en.wikipedia.org/wiki/Nazi_human_experimentation https://en.wikipedia.org/wiki/Nazi_human_experimentation I understand that certain people are salivating at the thought of a return to those times; I'm not one of them.
- smalltorch 3mo ago
  >Operation Sea-Spray was a 1950 U.S. Navy secret biological warfare experiment in which Serratia marcescens and Bacillus globigii bacteria were sprayed over the San Francisco Bay Area in California in order to determine how vulnerable a city like San Francisco would be to a bioweapon attack. There has been speculation that the experiment may have contributed to one death and at least ten illnesses. Wow.
- estearum 3mo ago
  Also, despite GP's dismissal of the "low-hanging fruit" hypothesis, it is obviously true that we've found the easy-to-discover drugs, and therefore drugs would get harder and harder to find. A flippant dismissal in an HN comment does not actually negate reality.
estearum 3mo ago
> Demands for increasingly byzantine trials This is silly. FDA has essentially one requirement: prove that your drug is safe and effective. The reason trial designs get more and more byzantine is because the drugs themselves work less-and-less well. They're far more nuanced and precise. The experiments have to be extremely well-controlled, and then this has to balance against cost/timeline of the trial, and that's why sponsors choose to use byzantine trial designs.
- A_D_E_P_T 3mo ago
  Proving "efficacy" -- which is the difficult and expensive part -- should not be necessary, and the government increasingly moves its own goalposts as to what the word efficacy even means. Simple as that. Postmarketing surveillance can easily determine what's effective and what's not, and medical orgs can adjust.
  - estearum 3mo ago
    Brilliant idea. This way both actual working drugs and vitamin-aisle potions look identical to consumers. Neither (or both?) can make claims to their effectiveness since they're both backed by the same (lack of) actual knowledge. This is especially great because it puts anyone who actually wants to actually make a working drug at a significant disadvantage. It'll take them longer to get to market, cost them a billion dollars more, then their medicine gets to sit next to a thousand variations of "Vitamin C for Leukemia" that all cost a lot less. There would be virtually no incentive for anyone to make an actual drug. > Postmarketing surveillance can easily determine what's effective and what's not, and medical orgs can adjust. "Easily" is doing a ton of work. Postmarketing surveillance can sometimes give low-confidence signal as to what's effective and what's not.
    - qsera 3mo ago
      >There would be virtually no incentive for anyone to make an actual drug. Dumb question: Wouldn't the market discover what works and what does not automatically, like any other product?
graemep 3mo ago
> Demands for increasingly byzantine trials have ballooned the costs associated with drug development, Even if that is true, is it an intrinsic problem with trials or just bad regulation? If it is the latter then you need to change the regulations? Is the problem global - is every regulator everywhere demanding byzantine trails?
- estearum 3mo ago
  Regulators don't demand byzantine trials. They'd prefer simpler ones for a million different reasons. Byzantine trial designs exist because it's incredibly hard to prove your drug works even in an RCT. But GP thinks you can just look at EMR data and ta-da now you know, lol.