5 ms·
I got the opposite impression. In fact, I think this may be the best medical statistics article I've ever seen in the popular press: But there was a problem
by nkurz 13y ago
I got the opposite impression. In fact, I think this may be the best medical statistics article I've ever seen in the popular press:
But there was a problem. This mild-to-moderate subgroup
wasn’t one the researchers said they would analyze when
they set up the study. Subdividing patients after the
fact and looking for statistically significant results
is a controversial practice. In its most extreme form,
it’s scorned as “data dredging.” The term suggests that
if you drag a net through a bunch of numbers enough
times, you’ll come up with something significant sooner
or later."
I don't presume it has anything to do with Bezos, but if it does, I hope stays on a buying spree! I was inspired to find out more about the author: http://www.washingtonpost.com/david-brown/2011/02/28/AB2Y0sM_page.html http://www.washingtonpost.com/david-brown/2011/02/28/AB2Y0sM...
It turns out he's a part-time journalist and part-time licensed physician: "He works four days a week at the Post and two-thirds of a day at a general internal medicine clinic in Baltimore supervising third-year medical students."
I also didn't find it biased toward Harkonen at all. Consider the closing:
InterMune did run another trial. It was big — 826
patients at 81 hospitals — in order to maximize the
chance of getting clear-cut results. It enrolled only
people with mild to moderate lung damage, the subgroup
whose success was touted in the press release.
And it failed. A little more than a year into the study,
more people on the drug had died (15 percent) than people
on placebo (13 percent). That was the death knell for the
drug.
It even links to the actual study:
http://www.ncbi.nlm.nih.gov/pubmed/19570573?dopt=Abstract http://www.ncbi.nlm.nih.gov/pubmed/19570573?dopt=Abstract
I don't know if I've ever seen a major newspaper link directly to a primary scientific paper as a source. I encourage you to read it again and see if your view changes. I've read it twice now and think it is an absolutely stellar piece of science writing, worthy of a Best Science Writing compilation.
- jonnathanson 13y agoI'll give it another read in light of what you're saying, but in the meantime, to clarify one line: "I don't presume it has anything to do with Bezos, but if it does, I hope stays on a buying spree!" I don't presume it has anything to do with Bezos, either. I would assume (?) he hasn't really taken the reins yet, at any rate. I've since edited that statement for clarity. Not trying to blame him for anything, or attribute this to him in any way. Rather, I am hopeful that the quality improves under him.
- paulftw 13y agoTone of the article was certainly skewed in Harkonen's favor, so I stopped mid-second page and turned to HN comments. It's great that the journalist found the debunking study and mentioned it. But does the order matter? He goes on and on about how "a sizable proportion of [scientists] might be in jail today". Most readers would drop half way through, as I did. What will they learn? Oh, there was evidence that drug can be lifesaving but technicality killed the white knight. The problems with after-the-fact analyses are well known. I reckon there should be a strict mathematical proof why those are wrong. We just got yet another example of that. How else can you keep charlatans out if not by being rough on those who break the rules? Especially when breaking the rules possibly has lethal consequences for less sophisticated yet desperate consumers. I'm not saying Harkonen is a charlatan, but he certainly jumped the gun. For whatever reason, but he did.
- freshhawk 13y agoSubdividing patients after the fact and looking for statistically significant results is a controversial practice If by controversial they meant "is considered outright fraud" ... then yes, it is controversial. This is one of the parts that made me see an unacceptable level of bias. There are a lot of parts that are similar to this, it might do a better than average job of explaining p-values and talking about the realities of experimental design but the tone in every case is dismissive in the "bunch of eggheads worrying about little details" kind of way. This is so biased I assume there is an agenda here rather than just bad journalism, the other explanation is that the writer is a good writer, researched the unfamiliar terms but didn't understand any of it and is good enough at regurgitating information to make it look like they did understand it. Either way, it's awful science writing even if they did use a lot of science words correctly. I have a higher standard for my hypothetical Best Science Writing awards I guess ... like requiring they get the science right.
- mbreese 13y ago> If by controversial they meant "is considered outright fraud" ... then yes, it is controversial Hold on there... controversial, yes, but fraud - no. If the original data was reported, and after the fact you discovered a class of patients responded differently than the rest and you could find a good classifier, then it's certainly not fraud. In this case, they weren't just taking all the "good" patients and putting them in one adhoc group and all the "bad" patients and putting them in another. They found they were able to sub-classify the patients using existing clinical traits. I thought the article did a pretty good job trying to explain the statistical issues at play. As described, the study isn't very clear, but it's hardly fraud to spin the results in that way. Is it reaching? Perhaps, but I so is calling this criminal. Remember using 0.05 as a p-value cutoff for significance is just a convention the number itself is arbitrary. In my work, I have to use something much more stringent such as 0.001 due to multiple testing issues. In my reading, the issue wasn't the analysis. The issue was the hype of the analysis. In this case, he definitely went overboard, but I'm not sure it rose to the level of fraud.
- PeterisP 13y agoIf you divide patients in 20+ arbitrary groups (say, based on first letter of their name) and give them a 0 efficiency treatment, then almost always you'll see a group with a p-0.05 significant improvement; that' just basic stats. So, if you afterwards discover some class of patients with this different response, it is grounds only to investigate that group further separately, not anything more - claiming that there is such an effect is fraud. You can't make extra claims until you have verification of that separate group. For an illustration, see http://xkcd.com/882/ http://xkcd.com/882/
- jonnathanson 13y agoI've reread the article a few times since reading your post, and while I might moderate the severity with which I first lashed at the author, I stand by my assessment. Here's my rationale: 1) The tone of the article is skewed in Harkonen's favor right from the outset, particularly in the way the author initially and repeatedly presumes the purity of Harkonen's motives ("saving lives"). In fact, we have no evidence of any motives other than the financial ("billion-dollar market," "$500 million opportunity," etc.). 2) Harkonen's methodology was flawed, full stop. While there might well have been something promising in the cohort he analyzed, a remainder cohort from one study does not take the place of a full set of clinical trials. Medical and statistical ethics would demand another study be conducted of the cohort Dr. Harkonen identified as more promising. Jumping the gun was unwarranted and possibly reckless. (As it turns out, a full study was later conducted, and indeed it proved the recklessness with which Harkonen leapt to his early conclusions). 3) Harkonen grossly misstated the results of the study in his press release, and, while it's hard to prove his intent, he is on record stating his financial upside in so doing. The author devotes all of a couple lines to this crucial point, then trots out a couple of irrelevant character-witness quotes to pave it over. (One quote essentially states 'If this is a crime, then a lot of scientists are in trouble!' Well, it is a crime. And if lots of other scientists are similarly cherrypicking their data and making misleading statements about it, then I'd say we have a serious problem on our hands.) 4) Your point about the closing statement in the article doesn't indicate absence of bias in favor of Harkonen. It adds some balance, yes. But it's buried at the bottom of the piece, where the typical reader either won't find it, or will glaze over it. 5) The opening paragraphs of the article, by contrast, paint a tragic portrait of an eminent and well-meaning doctor humiliated and punished for the crime of simply being overeager to help people (with the strong implication that his humiliation and punishment were unjust). 6) The author makes the repeated assertion that there is 'nothing factually inaccurate' about the content of Harkonen's press release. This is true by the letter, but not by the spirit. Harkonen cherrypicked the "facts" that he portrayed, accurately or otherwise, in that press release. And the later study bears out the potentially disastrous consequences of his having done so, negligently and recklessly. I'll grant you that "fraud" seems a little extreme in this case, and that's where the controversy lies. But we can demonstrate mens rea, and at the very least, Harkonen committed an act of criminal negligence. That he stood to gain significantly from his misleading statements is strong circumstantial evidence in favor of fraud, though the article doesn't give us enough facts to make a definitive claim there. At the very least, we see no evidence that Harkonen was doing this to save lives -- a point the author strongly and repeatedly implies throughout the article.