4 ms·

I'm not sure if I'm missing something either about statistics or about how the trials work, so maybe someone can enlighten me: > Based on patients’ ratings of

by corin_ 11y ago

I'm not sure if I'm missing something either about statistics or about how the trials work, so maybe someone can enlighten me: > Based on patients’ ratings of their pain, the effect of trialled drugs in relieving symptoms stayed the same over the 23-year period — but placebo responses rose. In 1996, patients in clinical trials reported that drugs relieved their pain by 27% more than did a placebo. But by 2013, that gap had slipped to just 9%. Wouldn't an easier way to look at it be to completely ignore the effects of the actual drugs in those trials, and just look at the data of how many people took placebos and to what extent those people reported changes?

lovemenot 11y ago
I have no insight, but looking at the text you highlighted, I expect they must have done exactly the analysis you suggest. It is likely that the researchers and the audience want to understand how to get more effective drugs onto the market more than why placebos per se are becoming relatively more effective. This may explain the unexpected reporting emphasis.
pdpi 11y ago
When you're doing a regular drug trial, you control using a placebo to account for the effect of just taking something. In this reverse analysis you need the opposite: if the placebo effect is changing, that could either be because the placebo itself is more effective, or because people are reporting pain differently, or a number of other confounding factors. Analysing the people who took real painkillers too gives you a baseline to compare against.
Phemist 11y ago
It's impossible to test "absolute" effects of drugs, so they are always tested in relation to each-other, this makes analysis easy because the only variable that's different is between whether participants are taking the actual drug, or a placebo. What you're suggesting would require testing a placebo in relation to not taking anything, which would introduce a lot more variables, and would make it impossible to do double-blind testing.
- corin_ 11y ago
  OK so the answer is I don't understand drug testing :) My understanding was: you need the placebo to compare to to counter out the placebo effect (i.e. if you only test "real drug vs. nothing", the real drug might show an improvement purely due to placebo and have no actual effect itself). But introducing more variables... surely there are lots of other variables there regardless? If you ran a test of "placebo vs. nothing" what variables come in that wouldn't be there for "placebo vs. drug"?
  - Phemist 11y ago
    First off, the reason double-blind testing is so effective is that it reduces subconscious biases in behaviour. If the person administering drugs knows its a placebo, he/she will consciously, or subconsciously, behave differently. All sorts of unwanted effects come into play, most importantly Observer-expectancy (For a good example, https://en.wikipedia.org/wiki/Clever_Hans https://en.wikipedia.org/wiki/Clever_Hans) Double-blind testing allows one to test the effects of administering anything (the placebo effect), versus the effects of the actual drugs. By administering an inert substance, the baseline placebo effect can be tested, as no actual medicative effects are expected from this inert substance. When testing placebo vs nothing, in the nothing case a lot changes. The participants will no longer be administered anything, the experimenters will be aware of not administering anything, etc. etc. Because so many variables change in such a set-up, you can no longer actually tell which of these variables changing caused the effect and which of them didn't. Thus making any results and conclusion derived from the suspect.
    - austinjp 11y ago
      I'm not entirely sure about that. Some trial designs do indeed have an arm of "no intervention" or "usual care" or "waiting". Sometimes these groups may be given the active therapy after the trial ends, since withholding a potentially effective and efficacious therapy may be considered unethical. There are many trial designs, not all of them are lab-based - - "pragmatic" trials in particular don't tend to follow typical masked, randomised, placebo-controlled designs.
jobigoud 11y ago
> In 1996, patients in clinical trials reported that drugs relieved their pain by 27% more than did a placebo. But by 2013, that gap had slipped to just 9%. Wait, this can be read as "Drugs are less efficient" just as well as "Placebo are more efficients".
- corin_ 11y ago
  They do say "Based on patients’ ratings of their pain, the effect of trialled drugs in relieving symptoms stayed the same over the 23-year period", too
  - saalweachter 11y ago
    Yeah, but when they compare subjective measurements across studies like that, it's really an apples and oranges comparison. The entire reason we use placebos is because there is no objective measurement for things like pain or sadness or "how well are you feeling in general?". If the gap between placebos and pain medications is narrowing, my first assumption is that the experimental designs have tightened over time, so that the blinding is better, and subjects and doctors are less able to distinguish between treatments. It's not that pain medication is "less" effective, it's that it was always only (say) 9% better than placebo, but in crappy earlier studies where it was less well blinded, some subjects (or researchers working with the subjects) could tell they were taking the placebo, and reported feeling worse, and some subjects could tell they were taking the treatment, and reported feeling better. If you tell me the gap has narrowed while the subjective reports of the treatments' effectiveness has stayed the same, I still believe my first assumption above, but now I'm a little bit suspicious of actual researcher bias. There are a vast number of ways researchers could "tune" for a target effect size, from classic ways like choosing the stopping point of the experiment to more subtle things like interpreting patient responses up or down based on whether a running analysis is coming in hot or cold. (The later could be less overtly malicious than you might think -- imagine that you're continually analyzing the results of your experiment, without breaking blinding. You "know", from past experiments, that treatment X is 70% effective. If you are seeing it as 80% effective in your running analysis, you might assume your interactions with your subjects are subtly biased to encourage over-reporting of the effect and consciously or unconsciously alter them to encourage under-reporting, and vice versa if you only see a 60% effect. If your blinding is still good, your gap in effectiveness will still narrow, but the "absolute effectiveness" of the treatment will still narrow.)

by corin_ 11y ago

lovemenot 11y ago

I have no insight, but looking at the text you highlighted, I expect they must have done exactly the analysis you suggest. It is likely that the researchers and the audience want to understand how to get more effective drugs onto the market more than why placebos per se are becoming relatively more effective. This may explain the unexpected reporting emphasis.

pdpi 11y ago

When you're doing a regular drug trial, you control using a placebo to account for the effect of just taking something. In this reverse analysis you need the opposite: if the placebo effect is changing, that could either be because the placebo itself is more effective, or because people are reporting pain differently, or a number of other confounding factors. Analysing the people who took real painkillers too gives you a baseline to compare against.

Phemist 11y ago

It's impossible to test "absolute" effects of drugs, so they are always tested in relation to each-other, this makes analysis easy because the only variable that's different is between whether participants are taking the actual drug, or a placebo. What you're suggesting would require testing a placebo in relation to not taking anything, which would introduce a lot more variables, and would make it impossible to do double-blind testing.

corin_ 11y ago
OK so the answer is I don't understand drug testing :) My understanding was: you need the placebo to compare to to counter out the placebo effect (i.e. if you only test "real drug vs. nothing", the real drug might show an improvement purely due to placebo and have no actual effect itself). But introducing more variables... surely there are lots of other variables there regardless? If you ran a test of "placebo vs. nothing" what variables come in that wouldn't be there for "placebo vs. drug"?
- Phemist 11y ago
  First off, the reason double-blind testing is so effective is that it reduces subconscious biases in behaviour. If the person administering drugs knows its a placebo, he/she will consciously, or subconsciously, behave differently. All sorts of unwanted effects come into play, most importantly Observer-expectancy (For a good example, https://en.wikipedia.org/wiki/Clever_Hans https://en.wikipedia.org/wiki/Clever_Hans) Double-blind testing allows one to test the effects of administering anything (the placebo effect), versus the effects of the actual drugs. By administering an inert substance, the baseline placebo effect can be tested, as no actual medicative effects are expected from this inert substance. When testing placebo vs nothing, in the nothing case a lot changes. The participants will no longer be administered anything, the experimenters will be aware of not administering anything, etc. etc. Because so many variables change in such a set-up, you can no longer actually tell which of these variables changing caused the effect and which of them didn't. Thus making any results and conclusion derived from the suspect.
  - austinjp 11y ago
    I'm not entirely sure about that. Some trial designs do indeed have an arm of "no intervention" or "usual care" or "waiting". Sometimes these groups may be given the active therapy after the trial ends, since withholding a potentially effective and efficacious therapy may be considered unethical. There are many trial designs, not all of them are lab-based - - "pragmatic" trials in particular don't tend to follow typical masked, randomised, placebo-controlled designs.

jobigoud 11y ago

> In 1996, patients in clinical trials reported that drugs relieved their pain by 27% more than did a placebo. But by 2013, that gap had slipped to just 9%. Wait, this can be read as "Drugs are less efficient" just as well as "Placebo are more efficients".

corin_ 11y ago
They do say "Based on patients’ ratings of their pain, the effect of trialled drugs in relieving symptoms stayed the same over the 23-year period", too
- saalweachter 11y ago
  Yeah, but when they compare subjective measurements across studies like that, it's really an apples and oranges comparison. The entire reason we use placebos is because there is no objective measurement for things like pain or sadness or "how well are you feeling in general?". If the gap between placebos and pain medications is narrowing, my first assumption is that the experimental designs have tightened over time, so that the blinding is better, and subjects and doctors are less able to distinguish between treatments. It's not that pain medication is "less" effective, it's that it was always only (say) 9% better than placebo, but in crappy earlier studies where it was less well blinded, some subjects (or researchers working with the subjects) could tell they were taking the placebo, and reported feeling worse, and some subjects could tell they were taking the treatment, and reported feeling better. If you tell me the gap has narrowed while the subjective reports of the treatments' effectiveness has stayed the same, I still believe my first assumption above, but now I'm a little bit suspicious of actual researcher bias. There are a vast number of ways researchers could "tune" for a target effect size, from classic ways like choosing the stopping point of the experiment to more subtle things like interpreting patient responses up or down based on whether a running analysis is coming in hot or cold. (The later could be less overtly malicious than you might think -- imagine that you're continually analyzing the results of your experiment, without breaking blinding. You "know", from past experiments, that treatment X is 70% effective. If you are seeing it as 80% effective in your running analysis, you might assume your interactions with your subjects are subtly biased to encourage over-reporting of the effect and consciously or unconsciously alter them to encourage under-reporting, and vice versa if you only see a 60% effect. If your blinding is still good, your gap in effectiveness will still narrow, but the "absolute effectiveness" of the treatment will still narrow.)