4 ms·

As a non-statistician I can tell you this blog is absolutely unconvincing, because it doesn't explain anything about what the test is. Apparently this test is

As a non-statistician I can tell you this blog is absolutely unconvincing, because it doesn't explain anything about what the test is. Apparently this test is how likely it is for the same number of people to have the primary diagnosis in each group because all the 1.0 have equal number of patients with that diagnosis or off by 1. Like if they said, we have 8 people with COPD in the first group and then waited for COPD patients to show up at the ER for the second group until they had 8 then you would get a perfect 1.0 match since they made that variable the same? And then the other variables would be randomish, but maybe correlated so more than 0.5? Maybe you can explain how this layman take is wrong, but if all that's going on is they selected patients with the same diagnoses and didn't make that clear in the paper then I don't understand what the big deal is.

jmount 5y ago
I'll give it a try. The study is two small populations that are assumed to be similar. If this were the case, then knowing the study ID (1 or 2) would sometimes wrongly look predictive for different conditions. However none of the measurements have this flaw, and that it self is very very unlikely. The claim is the data looks like it was designed to not have any imbalance from between sets 1 and 2, to an extend that is itself unlikely due to sampling.
- bsdetector 5y ago
  The entire second group was picked after the first one, and there were imbalances on almost half of the diagnoses. So if I understand this right, the problem is not the 1.0s since if they were selecting the same number of patients with the same primary diagnoses for the second group then those would be expected, but that there should be some 0.2s in there as well.
  - cvinn 5y ago
    For one thing, no matching was done (as the email explains).
    - bsdetector 5y ago
      "if they said, we have 8 people with COPD in the first group and then waited for COPD patients to show up at the ER for the second group until they had 8" Did you not read this? I asked whether if they did this they would come up with the perfect 1 scores on those items. Whether they actually did or not is a separate matter. What troubles me more than whether this study was faked or not is the certainty HN readers have that it was without being able to answer simple questions about why they believe that. Is this test just measuring the likelihood of each group having the same primary diagnosis? If they added people to the second group so they had the same number of a primary diagnosis, would that result in a 1.0 p value on this test? These should not be difficult questions to answer for somebody certain that fraud occurred.
      - civilized 5y ago
        1. The procedure you describe would make patient recruitment non-consecutive, contrary to the reported procedure. Consecutive means you don't skip anyone who has the condition you're trying to treat. You would have to skip people if you're selecting matched treatment and control groups. In this case, people with sepsis who don't fit your matching design have to be skipped and excluded from the study. 2. Patient subgroup counts matched perfectly not only on COPD, but on about a dozen other conditions. Difficulty of matching subgroup counts grows rapidly with number of dimensions. To match subgroup counts for Group A and Group B near-perfectly on a dozen different dimensions, there's no substantially easier method than just one-to-one perfect matching, which requires a very, very large number of patients. You have to take Patient A1, who has subset X1 of 12 different conditions, and find Patient B1 with that exact same subset. Then repeat, 47 times in this case. It is already quite hard to find the person B1 who has the exact subset X1 of conditions that person A1 had. For example, if there are 12 conditions, each condition is present in half the people that come into your clinic, and the conditions are independent, you'll need to go through 2^12 = 4096 people, on average, to find another exact match. The conditions may be a bit correlated but this can only help you so much when you're talking about 12 different conditions. To repeat that feat 47 times is very hard. You'd have to churn through 10s or 100s of thousands of sepsis patients to get your matching subset. This would require access to an enormous pool of sepsis patients and constant reporting of all the conditions they have, that you want to match on. For this to be done without any mention in the paper is utterly beyond belief. And the fact that the counts match near-perfectly in an off-by-1 fashion does not help the situation at all.
yyyk 5y ago
Am too a non-statistician. If I understand correctly: The two groups have very strong statistical relations across multiple variables (not just COPD), yet the study clearly states they were chosen almost randomly. This is extremly unlikely to happen in a perfectly randomized test. What you describe is selecting the group in advance so they match (though you use only one variable). There are cases where doing this selection is reasonable and is useful. However, the study describes a different selection process. If the selection process described was not the one actually used, what else in the study does not match reality?

jmount 5y ago

I'll give it a try. The study is two small populations that are assumed to be similar. If this were the case, then knowing the study ID (1 or 2) would sometimes wrongly look predictive for different conditions. However none of the measurements have this flaw, and that it self is very very unlikely. The claim is the data looks like it was designed to not have any imbalance from between sets 1 and 2, to an extend that is itself unlikely due to sampling.

bsdetector 5y ago
The entire second group was picked after the first one, and there were imbalances on almost half of the diagnoses. So if I understand this right, the problem is not the 1.0s since if they were selecting the same number of patients with the same primary diagnoses for the second group then those would be expected, but that there should be some 0.2s in there as well.
- cvinn 5y ago
  For one thing, no matching was done (as the email explains).
  - bsdetector 5y ago
    "if they said, we have 8 people with COPD in the first group and then waited for COPD patients to show up at the ER for the second group until they had 8" Did you not read this? I asked whether if they did this they would come up with the perfect 1 scores on those items. Whether they actually did or not is a separate matter. What troubles me more than whether this study was faked or not is the certainty HN readers have that it was without being able to answer simple questions about why they believe that. Is this test just measuring the likelihood of each group having the same primary diagnosis? If they added people to the second group so they had the same number of a primary diagnosis, would that result in a 1.0 p value on this test? These should not be difficult questions to answer for somebody certain that fraud occurred.
    - civilized 5y ago
      1. The procedure you describe would make patient recruitment non-consecutive, contrary to the reported procedure. Consecutive means you don't skip anyone who has the condition you're trying to treat. You would have to skip people if you're selecting matched treatment and control groups. In this case, people with sepsis who don't fit your matching design have to be skipped and excluded from the study. 2. Patient subgroup counts matched perfectly not only on COPD, but on about a dozen other conditions. Difficulty of matching subgroup counts grows rapidly with number of dimensions. To match subgroup counts for Group A and Group B near-perfectly on a dozen different dimensions, there's no substantially easier method than just one-to-one perfect matching, which requires a very, very large number of patients. You have to take Patient A1, who has subset X1 of 12 different conditions, and find Patient B1 with that exact same subset. Then repeat, 47 times in this case. It is already quite hard to find the person B1 who has the exact subset X1 of conditions that person A1 had. For example, if there are 12 conditions, each condition is present in half the people that come into your clinic, and the conditions are independent, you'll need to go through 2^12 = 4096 people, on average, to find another exact match. The conditions may be a bit correlated but this can only help you so much when you're talking about 12 different conditions. To repeat that feat 47 times is very hard. You'd have to churn through 10s or 100s of thousands of sepsis patients to get your matching subset. This would require access to an enormous pool of sepsis patients and constant reporting of all the conditions they have, that you want to match on. For this to be done without any mention in the paper is utterly beyond belief. And the fact that the counts match near-perfectly in an off-by-1 fashion does not help the situation at all.

yyyk 5y ago

Am too a non-statistician. If I understand correctly: The two groups have very strong statistical relations across multiple variables (not just COPD), yet the study clearly states they were chosen almost randomly. This is extremly unlikely to happen in a perfectly randomized test. What you describe is selecting the group in advance so they match (though you use only one variable). There are cases where doing this selection is reasonable and is useful. However, the study describes a different selection process. If the selection process described was not the one actually used, what else in the study does not match reality?