3 ms·

Subdividing patients after the fact and looking for statistically significant results is a controversial practice If by controversial they meant "is consid

Subdividing patients after the fact and looking for statistically significant results is a controversial practice If by controversial they meant "is considered outright fraud" ... then yes, it is controversial. This is one of the parts that made me see an unacceptable level of bias. There are a lot of parts that are similar to this, it might do a better than average job of explaining p-values and talking about the realities of experimental design but the tone in every case is dismissive in the "bunch of eggheads worrying about little details" kind of way. This is so biased I assume there is an agenda here rather than just bad journalism, the other explanation is that the writer is a good writer, researched the unfamiliar terms but didn't understand any of it and is good enough at regurgitating information to make it look like they did understand it. Either way, it's awful science writing even if they did use a lot of science words correctly. I have a higher standard for my hypothetical Best Science Writing awards I guess ... like requiring they get the science right.

mbreese 13y ago
> If by controversial they meant "is considered outright fraud" ... then yes, it is controversial Hold on there... controversial, yes, but fraud - no. If the original data was reported, and after the fact you discovered a class of patients responded differently than the rest and you could find a good classifier, then it's certainly not fraud. In this case, they weren't just taking all the "good" patients and putting them in one adhoc group and all the "bad" patients and putting them in another. They found they were able to sub-classify the patients using existing clinical traits. I thought the article did a pretty good job trying to explain the statistical issues at play. As described, the study isn't very clear, but it's hardly fraud to spin the results in that way. Is it reaching? Perhaps, but I so is calling this criminal. Remember using 0.05 as a p-value cutoff for significance is just a convention the number itself is arbitrary. In my work, I have to use something much more stringent such as 0.001 due to multiple testing issues. In my reading, the issue wasn't the analysis. The issue was the hype of the analysis. In this case, he definitely went overboard, but I'm not sure it rose to the level of fraud.
- PeterisP 13y ago
  If you divide patients in 20+ arbitrary groups (say, based on first letter of their name) and give them a 0 efficiency treatment, then almost always you'll see a group with a p-0.05 significant improvement; that' just basic stats. So, if you afterwards discover some class of patients with this different response, it is grounds only to investigate that group further separately, not anything more - claiming that there is such an effect is fraud. You can't make extra claims until you have verification of that separate group. For an illustration, see http://xkcd.com/882/ http://xkcd.com/882/
  - mbreese 13y ago
    Of course, but they divided the patients into two groups: Mild/moderate and severe clinical phenotypes. It's not optimal, but it is far from subdividing into 20 classes or making up arbitrary classifiers until you got the number you were looking for. The clinical phenotypes were already established prior to the analysis.
- freshhawk 13y ago
  > and after the fact you discovered a class of patients responded differently than the rest and you could find a good classifier, then it's certainly not fraud Or you could say "after the fact you chose/created the class of patients that responded differently". It certainly is fraudulent, I suppose this issue isn't quite a Stats 101 thing, but it's certainly undergrad level because it's only a little bit subtle. A useful metaphor to keep in mind is the one that gives the Texas Sharpshooter Fallacy it's name: "The name comes from a joke about a Texan who fires some shots at the side of a barn, then paints a target centered on the biggest cluster of hits and claims to be a sharpshooter." The examples on wikipedia are also instructive (http://en.wikipedia.org/wiki/Texas_sharpshooter_fallacy http://en.wikipedia.org/wiki/Texas_sharpshooter_fallacy) As PeterisP pointed out, the only correct interpretation is to say "That's interesting, we need to do another study with that subgroup to see if that's a real effect" (although that xkcd comic illustrates what's called "The Filedrawer Effect" or simply "publication bias" which is the same error made a different way). Give me a dataset of people rolling dice many times and the ability to group them after examining the data and I will find you a group that seems to have magical craps playing abilities that don't actually exist. It might be men over 6' with brown hair or it might be married people who are left handed or maybe bilingual introverts ... who knows, I can't tell you what that group will be until I see the data (and that last statement is why you know it's not a real ability). This is precisely why experimental design depends on defining your tests before you examine your data or, put another way, you cannot use the data that suggested an effect to prove an effect. The idea that the CEO of a biotech company didn't know this is absurd, because this is Running a Scientific Experiment 101. Selecting groups after seeing the data is probably the most common method chosen to tweak failed experiments by those committing fraud, second is probably using the filedrawer (just rerun the experiment until you get a result by chance and pretend the failed experiments never happened). These are so rife that there are hundreds of articles about the prevalence of this type of fraud published in the last few years. This type of behaviour is a big deal these days.

mbreese 13y ago

> If by controversial they meant "is considered outright fraud" ... then yes, it is controversial Hold on there... controversial, yes, but fraud - no. If the original data was reported, and after the fact you discovered a class of patients responded differently than the rest and you could find a good classifier, then it's certainly not fraud. In this case, they weren't just taking all the "good" patients and putting them in one adhoc group and all the "bad" patients and putting them in another. They found they were able to sub-classify the patients using existing clinical traits. I thought the article did a pretty good job trying to explain the statistical issues at play. As described, the study isn't very clear, but it's hardly fraud to spin the results in that way. Is it reaching? Perhaps, but I so is calling this criminal. Remember using 0.05 as a p-value cutoff for significance is just a convention the number itself is arbitrary. In my work, I have to use something much more stringent such as 0.001 due to multiple testing issues. In my reading, the issue wasn't the analysis. The issue was the hype of the analysis. In this case, he definitely went overboard, but I'm not sure it rose to the level of fraud.

PeterisP 13y ago
If you divide patients in 20+ arbitrary groups (say, based on first letter of their name) and give them a 0 efficiency treatment, then almost always you'll see a group with a p-0.05 significant improvement; that' just basic stats. So, if you afterwards discover some class of patients with this different response, it is grounds only to investigate that group further separately, not anything more - claiming that there is such an effect is fraud. You can't make extra claims until you have verification of that separate group. For an illustration, see http://xkcd.com/882/ http://xkcd.com/882/
- mbreese 13y ago
  Of course, but they divided the patients into two groups: Mild/moderate and severe clinical phenotypes. It's not optimal, but it is far from subdividing into 20 classes or making up arbitrary classifiers until you got the number you were looking for. The clinical phenotypes were already established prior to the analysis.
freshhawk 13y ago
> and after the fact you discovered a class of patients responded differently than the rest and you could find a good classifier, then it's certainly not fraud Or you could say "after the fact you chose/created the class of patients that responded differently". It certainly is fraudulent, I suppose this issue isn't quite a Stats 101 thing, but it's certainly undergrad level because it's only a little bit subtle. A useful metaphor to keep in mind is the one that gives the Texas Sharpshooter Fallacy it's name: "The name comes from a joke about a Texan who fires some shots at the side of a barn, then paints a target centered on the biggest cluster of hits and claims to be a sharpshooter." The examples on wikipedia are also instructive (http://en.wikipedia.org/wiki/Texas_sharpshooter_fallacy http://en.wikipedia.org/wiki/Texas_sharpshooter_fallacy) As PeterisP pointed out, the only correct interpretation is to say "That's interesting, we need to do another study with that subgroup to see if that's a real effect" (although that xkcd comic illustrates what's called "The Filedrawer Effect" or simply "publication bias" which is the same error made a different way). Give me a dataset of people rolling dice many times and the ability to group them after examining the data and I will find you a group that seems to have magical craps playing abilities that don't actually exist. It might be men over 6' with brown hair or it might be married people who are left handed or maybe bilingual introverts ... who knows, I can't tell you what that group will be until I see the data (and that last statement is why you know it's not a real ability). This is precisely why experimental design depends on defining your tests before you examine your data or, put another way, you cannot use the data that suggested an effect to prove an effect. The idea that the CEO of a biotech company didn't know this is absurd, because this is Running a Scientific Experiment 101. Selecting groups after seeing the data is probably the most common method chosen to tweak failed experiments by those committing fraud, second is probably using the filedrawer (just rerun the experiment until you get a result by chance and pretend the failed experiments never happened). These are so rife that there are hundreds of articles about the prevalence of this type of fraud published in the last few years. This type of behaviour is a big deal these days.