4 ms·

As a trained (and hopefully ethical) statistician, I agree with the interpretation that Dr. Harkonen did mis-represent the results of the trial. This article do

As a trained (and hopefully ethical) statistician, I agree with the interpretation that Dr. Harkonen did mis-represent the results of the trial. This article does a pretty good job describing the controversy to a layperson audience, but I do not feel the result is all that controversial. It is well-known to statistically-minded people that p-values are computations that rely on particular assumptions being made. One of those assumptions is that only a single, pre-specified hypothesis is being tested. By making additional comparisons, the p-value that was reported by Dr. Harkonen mis-represented the true significance of the trial. Perhaps factually the p-value was 0.004, but publicizing the p-value as if it were obtained by a fair trial, as opposed to finding it in a hunt for (quasi-)statistical significance, was a manipulation of the facts to support one's personal interests. That's not science; it's bias, and self-serving bias at that.

mbreese 13y ago
I'm not as convinced as you are. Just because the original study wasn't found to be significant, that doesn't mean that an already-existing sub group wasn't significant. They used an existing clinical trait as a separate classifier to look at the patients in a different way. That isn't too controversial (if you have a high enough patient count, which was probably the biggest fault of this post-hoc analysis). Then again, I'm a biologist. We're trained to not trust statisticians. (Or course, we're also trained to not speak in absolutes and cover every statement in doubt, so he failed in that regard too).
- dalek_cannes 13y ago
  I initially thought he was partially justified in claiming the number with the 'mild-to-medium patients' qualification if there was a real benefit in the drug that the study had not been designed to detect. Except that a follow up study focusing on the mild-to-medium subgroup failed to show statistically significant benefits.
- jerrytsai 13y ago
  What is likely, although we cannot of course know for sure, is that the doctor looked at more than disease severity (the "existing clinical trait") as a separate classifier, hunting for subgroups for which the p-value indicated a promising trend. The principle behind the proscription against multiple comparisons is well-known to statisticians. If we consider a 1 chance in 20 result to be statistically significant, then, randomly, on average 20 "trials" will yield one statistically significant result. By dividing the patients into disease severity subgroups, Dr. Harkonen increased the number of "trials" from 1 to 4, thereby elevating the likelihood of yielding an effect that appeared to be statistically significant. If he also examined other subgroups in his quest to find a positive result, then he elevated the likelihood of finding a positive result toward certainty. Our desire to find patterns and see cause and effect make us prone to confirmation bias. We can guard against this bias with care, including the use of statistics. It was not a surprise that a subsequent study looking only at the "mild-to-moderate" group did not demonstrate any benefit of the treatment. The belief that the treatment would benefit "mild-to-moderate" patients was speciously derived.
  - mbreese 13y ago
    Right... multiple testing correction, false-discovery rates, etc... I'm quite familiar. (Even if they did correct for multiple tests, I think the sub-group would have potentially been significant. An uncorrected p-value of 0.004 is what is sticking in my head). But the point is, just because he's bad at statistics does that make it fraud? Based on what we know from the article, I'd argue no. People are allowed to be wrong and make mistakes in their analysis. They just aren't allowed to knowingly make those mistakes. And this is what we don't know... what he knew and what he thought at the time.
    - x0x0 13y ago
      sure, but when people deliberately lie in order to gain millions of dollars, I can't get that upset when they get 6 months stuck in their house he, and statisticians at his company, knew or should have known what he was doing was wrong. This stuff is covered in the first inferential stats course taken as an undergrad.

mbreese 13y ago

I'm not as convinced as you are. Just because the original study wasn't found to be significant, that doesn't mean that an already-existing sub group wasn't significant. They used an existing clinical trait as a separate classifier to look at the patients in a different way. That isn't too controversial (if you have a high enough patient count, which was probably the biggest fault of this post-hoc analysis). Then again, I'm a biologist. We're trained to not trust statisticians. (Or course, we're also trained to not speak in absolutes and cover every statement in doubt, so he failed in that regard too).

dalek_cannes 13y ago
I initially thought he was partially justified in claiming the number with the 'mild-to-medium patients' qualification if there was a real benefit in the drug that the study had not been designed to detect. Except that a follow up study focusing on the mild-to-medium subgroup failed to show statistically significant benefits.
jerrytsai 13y ago
What is likely, although we cannot of course know for sure, is that the doctor looked at more than disease severity (the "existing clinical trait") as a separate classifier, hunting for subgroups for which the p-value indicated a promising trend. The principle behind the proscription against multiple comparisons is well-known to statisticians. If we consider a 1 chance in 20 result to be statistically significant, then, randomly, on average 20 "trials" will yield one statistically significant result. By dividing the patients into disease severity subgroups, Dr. Harkonen increased the number of "trials" from 1 to 4, thereby elevating the likelihood of yielding an effect that appeared to be statistically significant. If he also examined other subgroups in his quest to find a positive result, then he elevated the likelihood of finding a positive result toward certainty. Our desire to find patterns and see cause and effect make us prone to confirmation bias. We can guard against this bias with care, including the use of statistics. It was not a surprise that a subsequent study looking only at the "mild-to-moderate" group did not demonstrate any benefit of the treatment. The belief that the treatment would benefit "mild-to-moderate" patients was speciously derived.
- mbreese 13y ago
  Right... multiple testing correction, false-discovery rates, etc... I'm quite familiar. (Even if they did correct for multiple tests, I think the sub-group would have potentially been significant. An uncorrected p-value of 0.004 is what is sticking in my head). But the point is, just because he's bad at statistics does that make it fraud? Based on what we know from the article, I'd argue no. People are allowed to be wrong and make mistakes in their analysis. They just aren't allowed to knowingly make those mistakes. And this is what we don't know... what he knew and what he thought at the time.
  - x0x0 13y ago
    sure, but when people deliberately lie in order to gain millions of dollars, I can't get that upset when they get 6 months stuck in their house he, and statisticians at his company, knew or should have known what he was doing was wrong. This stuff is covered in the first inferential stats course taken as an undergrad.