4 ms·

They do say "Based on patients’ ratings of their pain, the effect of trialled drugs in relieving symptoms stayed the same over the 23-year period", too

by corin_ 11y ago

saalweachter 11y ago
Yeah, but when they compare subjective measurements across studies like that, it's really an apples and oranges comparison. The entire reason we use placebos is because there is no objective measurement for things like pain or sadness or "how well are you feeling in general?". If the gap between placebos and pain medications is narrowing, my first assumption is that the experimental designs have tightened over time, so that the blinding is better, and subjects and doctors are less able to distinguish between treatments. It's not that pain medication is "less" effective, it's that it was always only (say) 9% better than placebo, but in crappy earlier studies where it was less well blinded, some subjects (or researchers working with the subjects) could tell they were taking the placebo, and reported feeling worse, and some subjects could tell they were taking the treatment, and reported feeling better. If you tell me the gap has narrowed while the subjective reports of the treatments' effectiveness has stayed the same, I still believe my first assumption above, but now I'm a little bit suspicious of actual researcher bias. There are a vast number of ways researchers could "tune" for a target effect size, from classic ways like choosing the stopping point of the experiment to more subtle things like interpreting patient responses up or down based on whether a running analysis is coming in hot or cold. (The later could be less overtly malicious than you might think -- imagine that you're continually analyzing the results of your experiment, without breaking blinding. You "know", from past experiments, that treatment X is 70% effective. If you are seeing it as 80% effective in your running analysis, you might assume your interactions with your subjects are subtly biased to encourage over-reporting of the effect and consciously or unconsciously alter them to encourage under-reporting, and vice versa if you only see a 60% effect. If your blinding is still good, your gap in effectiveness will still narrow, but the "absolute effectiveness" of the treatment will still narrow.)

They do say "Based on patients’ ratings of their pain, the effect of trialled drugs in relieving symptoms stayed the same over the 23-year period", too

by corin_ 11y ago

They do say "Based on patients’ ratings of their pain, the effect of trialled drugs in relieving symptoms stayed the same over the 23-year period", too

saalweachter 11y ago

Yeah, but when they compare subjective measurements across studies like that, it's really an apples and oranges comparison. The entire reason we use placebos is because there is no objective measurement for things like pain or sadness or "how well are you feeling in general?". If the gap between placebos and pain medications is narrowing, my first assumption is that the experimental designs have tightened over time, so that the blinding is better, and subjects and doctors are less able to distinguish between treatments. It's not that pain medication is "less" effective, it's that it was always only (say) 9% better than placebo, but in crappy earlier studies where it was less well blinded, some subjects (or researchers working with the subjects) could tell they were taking the placebo, and reported feeling worse, and some subjects could tell they were taking the treatment, and reported feeling better. If you tell me the gap has narrowed while the subjective reports of the treatments' effectiveness has stayed the same, I still believe my first assumption above, but now I'm a little bit suspicious of actual researcher bias. There are a vast number of ways researchers could "tune" for a target effect size, from classic ways like choosing the stopping point of the experiment to more subtle things like interpreting patient responses up or down based on whether a running analysis is coming in hot or cold. (The later could be less overtly malicious than you might think -- imagine that you're continually analyzing the results of your experiment, without breaking blinding. You "know", from past experiments, that treatment X is 70% effective. If you are seeing it as 80% effective in your running analysis, you might assume your interactions with your subjects are subtly biased to encourage over-reporting of the effect and consciously or unconsciously alter them to encourage under-reporting, and vice versa if you only see a 60% effect. If your blinding is still good, your gap in effectiveness will still narrow, but the "absolute effectiveness" of the treatment will still narrow.)