3 ms·

The idea of rigging a clinical protocol to titrate out placebo responders and redo the study on successively more favorable terms seems like a new variety of in

by uvdiv 14y ago

The idea of rigging a clinical protocol to titrate out placebo responders and redo the study on successively more favorable terms seems like a new variety of intellectual corruption. They're not more favorable terms. A drug which has no effect will still perform identically to a placebo. What changes is the level of noise, and hence the sensitivity of the test. The spread in placebo responses (the standard deviation of the binomial distribution) is sqrt(N p (1-p)). For N=100, if 20% of the sample has a placebo response, the s.d. of this is 4%. If you have only 5% placebo responses, it's half that: 2.2%. Higher signal-to-noise ratio. You find that more drugs work, because it's easier to tell if they work -- not because you're corrupting the test to show that work when they actually don't.

gruseom 14y ago
Sorry, I'm not getting it—it sounds like tampering with a random sample after the fact. How does this not amount to throwing out data one doesn't like? Suppose I have a stock trading algorithm. I test it on random stocks, discard part of the data set, and run the tests again. Now it's easier to show that my system works. Is this ok? You find that more drugs work, because it's easier to tell if they work Previously, the definition of "works" was "can be shown to be significantly better than placebo". Why should that definition be changed? It seems very reasonable to me, whereas relabeling placebo effects "noise" to throw them out seems like eliminating the competition.
- uvdiv 14y ago
  Previously, the definition of "works" was "can be shown to be significantly better than placebo" That's what they're testing -- except, on a subgroup which is less sensitive to placebos. If a drug works on this subgroup, and if (the assumption) the drug effect is independent of the placebo effect, then you infer it's better than a placebo on the whole population, though you haven't directly tested this. Placebo effects don't vanish when you're given a real drug. Roughly, when you're testing a drug against a placebo, you're measuring {drug effect + placebo effect} against {placebo effect}. If drug and placebo are independent, than subtracting some "placebo effect" from both sides, equally, gives you the same comparison except with less statistical variation -- less "noise". It's still possible that the drug and placebo effect are interacting in an unknown way -- if the drug works, and separately it weakens the placebo effect, then this test gives you the wrong inference. This adds some epistemic uncertainty, though maybe less uncertainty than the statistical uncertainty in the ordinary trials. Obviously an issue. What this is isn't is biased. It's not set up in a way to make drugs seem systematically better: it's set up to find more drugs that actually are better.
  - gruseom 14y ago
    This thinking seems to me fraught with dangerous non sequiturs, or at least unproven assumptions. You mention one: on what basis can we assume that the drug effect and the placebo effect are independent? We don't understand the placebo effect. We should make as few assumptions about it as possible. Here is another. If the drug is approved, it won't be given only to the "subgroup which is less sensitive to placebos" on which it was tested. It will be given to the general population. On what basis can you assume that since the drug beats placebo on the subgroup, it must also beat placebo on the general population? It seems weird to allow a random sample to be taken from a population that is not the population you're trying to make claims about. Are there precedents for this in other fields?

by uvdiv 14y ago

gruseom 14y ago

Sorry, I'm not getting it—it sounds like tampering with a random sample after the fact. How does this not amount to throwing out data one doesn't like? Suppose I have a stock trading algorithm. I test it on random stocks, discard part of the data set, and run the tests again. Now it's easier to show that my system works. Is this ok? You find that more drugs work, because it's easier to tell if they work Previously, the definition of "works" was "can be shown to be significantly better than placebo". Why should that definition be changed? It seems very reasonable to me, whereas relabeling placebo effects "noise" to throw them out seems like eliminating the competition.

uvdiv 14y ago
Previously, the definition of "works" was "can be shown to be significantly better than placebo" That's what they're testing -- except, on a subgroup which is less sensitive to placebos. If a drug works on this subgroup, and if (the assumption) the drug effect is independent of the placebo effect, then you infer it's better than a placebo on the whole population, though you haven't directly tested this. Placebo effects don't vanish when you're given a real drug. Roughly, when you're testing a drug against a placebo, you're measuring {drug effect + placebo effect} against {placebo effect}. If drug and placebo are independent, than subtracting some "placebo effect" from both sides, equally, gives you the same comparison except with less statistical variation -- less "noise". It's still possible that the drug and placebo effect are interacting in an unknown way -- if the drug works, and separately it weakens the placebo effect, then this test gives you the wrong inference. This adds some epistemic uncertainty, though maybe less uncertainty than the statistical uncertainty in the ordinary trials. Obviously an issue. What this is isn't is biased. It's not set up in a way to make drugs seem systematically better: it's set up to find more drugs that actually are better.
- gruseom 14y ago
  This thinking seems to me fraught with dangerous non sequiturs, or at least unproven assumptions. You mention one: on what basis can we assume that the drug effect and the placebo effect are independent? We don't understand the placebo effect. We should make as few assumptions about it as possible. Here is another. If the drug is approved, it won't be given only to the "subgroup which is less sensitive to placebos" on which it was tested. It will be given to the general population. On what basis can you assume that since the drug beats placebo on the subgroup, it must also beat placebo on the general population? It seems weird to allow a random sample to be taken from a population that is not the population you're trying to make claims about. Are there precedents for this in other fields?