4 ms·
> ...I have not requested access to the raw data or contacted the authors for explanation as the case is audacious... Happily from the sidelines - it probably
by roenxi 5y ago
> ...I have not requested access to the raw data or contacted the authors for explanation as the case is audacious...
Happily from the sidelines - it probably is fraud and this isn't really an article targeted at the general audience. But the statistical evidence proves beyond a doubt that the data wasn't collected by the methods described in the paper. That doesn't prove it is fraudulent because formally speaking he can't rule out that they just did a very bad job of explaining their methods.
Just from life experience, any time anyone goes in guns blazing based on statistical evidence - without actually talking to the people involved - it is often a mistake. Just because someone can't think of an alternative reason says more about their imagination than the reality of the situation. Particularly coming in hot off a twitter argument.
That being said, these are unusual and unexpected patterns that show irregularities in the data, and I would certainly share his concerns.
- ai_ja_nai 5y agoI'd like to question how poorly the article was peer reviewed before publication
- JoachimS 5y agoYou need to consider the historical context too. The referenced authors has been found performing statistical fraud before. P(Mistake) is probably lower.
- ImaCake 5y agoNo there is no reason to be skeptical of the author here because he based his evidence on statistics. As the author describes, the p-values listed are effectively impossible to get from real data. Because of the magic of mathematics, we know exactly how p-values are meant to behave when you calculate them repeatedly - uniform random between 0 and 1. There are a battery of tests you can use to interrogate this very well understood distribution. Those tests, as the author states, prove without any doubt that the p-values in table 1 are effectively impossible. You can't make it to those p-values by mistake.
- roenxi 5y agoP values are, effectively, the probability of a result occurring because of a process. These p values are overwhelming evidence that the result did not occur because of the process described in the study. That does not mean fraud (although, I stress, in this case I can see why it is probably fraud). For example, maybe the study is done in some sort of hospital with some sort of patient scheduling system and the front desk is playing a funny joke on the doctors in this study. It is quite common for experimenters to not be as much in control of their environment as a paper suggests - that would be an honest reason for why the data is weird. In a more plausible example - I don't know how "matching" works, but maybe the patients were matched, the report writer didn't know that and the sentence slipped through proofreading. Unlikely, but certainly possible. That mistake would happen in the wild from time to time. You don't need much imagination to find examples other than fraud. Although it does take a little imagination, which is concerning. But, frankly, there should almost always be an attempt at dialog with someone before accusing them publicly of being a fraud. Unless they're going to shoot you or something extreme.
- hackernewds 5y agoTo be frank the instances you listed sound very improbable, but could withstand the burden of truth needed to prove guilty
- fieldcny 5y agoThe author of this work addressed the notion that patients were matched and the methods were described incorrectly, even in that situation the distribution of p values is so highly unlikely that it’s most likely fraud. Sloppy research methods leading to bogus results are can be construed as fraud, particularly if you don’t know the motivation for the lack of care. A diligent researcher might take these results which were “incredible” and attempt to replicate them before publishing.
- roenxi 5y agoSheldrick has just caused massive reputational damage, got thousands of strangers reading & discussing how Marik's work looks like fraud (despite few of us actually being qualified to assess its merits) & is posing a serious professional attack to Marik's wellbeing. It is unlikely but in the realm of possibility that someone would commit suicide or get some sort of trauma from this sort of attention. He's doing that based on an interpretation of some study, which I suspect is approximately a 20 page document. Now on the one hand, he is probably right. On the other hand, this is a very brave approach - the damage and the care taken aren't in good proportion here. This sort of thing is easy to get wrong. He should have talked to the paper's authors first.
- civilized 5y agoNo. Absolutely not. There is no non-fraudulent explanation here. In the best case of "poorly described methods", we're talking about authors who blatantly lied about how they sampled their patients and deliberately omitted what would have been an extremely laborious process, involving many thousands of extremely non-consecutive patients, to produce an extremely well-matched treatment and control group. You've got your choice as to the exact nature of the enormous lie or fabrication, but there's no possibility of innocent omission.
- radford-neal 5y agoI think it's difficult to eliminate the possibility of a tremendous degree of incompetence, without outright fraud, based just on the p-values near 1. As noted in other comments here, one possibility is that the authors failed to correctly describe the procedure they used. Sheldrick does consider this, pointing out that even if they actually did try to match subjects, unlike what they said in the paper, it would be very unlikely that they could match them well enough to obtain the reported p-values. But another possibility is that they were totally incompetent at using their statistical software - that the reported p-values are not actually the real p-values for the tests that they said they did. Maybe they accidently used the wrong dataset, for example. Or perhaps they copied numbers from the wrong column of the table the software produced. In a wider sense, though, this would still essentially be fraud - it's dishonest to publish a paper that purports to be a competent study when really you are totally clueless about how to do research, or care so little about correctness that you put no effort into checking that you did things correctly. (And surely they would have to suspect that they don't really know what they're doing...)
- civilized 5y agoThis isn't about reported p-values. Sheldrick computed p-values from the reported treatment and control group samples. The latter are basic count data, not the output of statistical software. So here we are talking about either the raw data being wrong, or this tiny amount of raw data being incorrectly summarized. Something like a column duplication would have resulted in perfectly identical data between treatment and control groups, which this data does not show. The best case scenario is an errant copy-paste or Excel formula that somehow presented both groups of data in a way that actually only looked at one of the two groups, but also somehow introduced small differences along the way. How exactly this could happen is beyond my imagination, but perhaps it introduces a sliver of doubt as to whether the paper is fraud by intention or merely by negligence.