3 ms·

This study mananaged to finance 38 participants worth of MRI time and (likely significant) monetary compensation for all 132 participants. Yet it wastes no more

• godelski 3y ago
  For anyone else who didn't RTFA, here's the direct quote. Seems reasonable to me... ---------------------- 2.5. Impact of COVID-19 pandemic Due to the COVID-19 pandemic, the UCI campus was closed in April 2020, and remained closed until the Fall of 2020. In addition, many participants did not feel comfortable entering the campus due to COVID-19 concerns even after the campus was officially open. As a result, participants who would have completed their 6-months of participation after April 2020 were either not able to return or chose not to return to campus for their second assessment. During the campus shutdown, contact was maintained with the participants who were impacted, and they were encouraged to continue their sensory enrichment, however, compliance was variable. When it became clear that the campus was going to remain closed for an extended period, we developed methods to remotely conduct the cognitive assessments using videoconferencing (Zoom app). When the campus re-opened and research participants were allowed back onto campus, participants who had received MRI scans at baseline received their second MRI scan. The data set used for the cognitive assessment analysis was reduced due to a number of possible confounding issues including the different conditions present for the cognitive assessment testing at baseline (in office) and that given remotely (in their home using videoconferencing), the possible sensitivity of that testing to the immediate physical environment during the assessment, as well as the variable timing both between the date of the baseline assessment and the date of final assessment, and the date of their final assessment and the date they discontinued their sensory enrichment. Accordingly, in our data analysis for cognitive assessment, we only included individuals who had completed their 6-months of participation prior to the UCI shutdown (a total of 11 controls and 12 enriched). For the MRI analysis, we included everyone who returned to campus for their follow-up MRI despite the difference in time (range: 6–17 months; a total of 23 controls and 20 enriched).
  • miggol 3y ago
    Unfortunately, reasonable has nothing to do with it. Though I must admit my first comment is rather sarcastic and not very informative. Let me elaborate on why the high drop out rate is so damning. Let's use a marble analogy because they are common in statistics. You gather 132 marbles of that vary in size in a way that is representative of the marble population as a whole. You then randomly assign them to either an intervention (n=68) or control (n=64) group. This random assignment is already an intervention in its own right, but with these numbers you could still say the groups are pretty much comparable. Now, you run your experiment. Your hypothesis is that the marbles in your intervention group will grow in size, and not the control. But you don't know that, and we should assume the null hypothesis until disproven. It's time to measure your marbles. It would make sense to measure them all, but you don't do that. You take a sample (n=12) for the intervention group and another sample from the control (n=11) and measure those samples. The rest are dropouts. The reasons that caused these marbles to drop out are irrelevant. I'm not suggesting the researchers are paid shills and consciously selected marbles to suit their narrative. For all we know all the other marbles were stolen, or are invalid for other causes outside the researchers' control. The problem is that this act of subsampling has wrecked the experimental design either way. The null hypothesis suggests that both groups were not only initially equal, but also remained equal regardless of intervention. So you must ask yourself the following: if I took such a small sample from each group before I had even performed the intervention, what are the odds that one sample would be significantly bigger than the other by pure chance? I'm not doing the maths, and they would heavily rely on the imaginary distribution of marble sizes in this analogy anyway. But I dare say that one sample would be significantly bigger than the other one often enough. And half of those times it would be the intervention group. So before even considering the effects of your own intervention, you're already fighting a losing battle. Because of subsampling the null hypothesis has grown stronger and can now explain even a very large difference between groups. So what do you do as a conscientious researcher when faced with the hardships of campus shutdowns and participants that didn't do cognitive tests in person but via zoom? My recommendation would be to run your statistical tests on everything and publish it anyway. Acknowledge that garbage in = garbage out, there were many confounders, so you can't draw any conclusions. Sometimes pandemics get in the way of science. You could even publish the results for all participants side by side with the restrictive subsample. If they both point to the same result, perhaps you even have grounds for some kind of valid conclusion! Makes you wonder why they didn't just do that, huh. What you definitely shouldn't do is only publish your statistical tests on the subgroup and draw conclusions based on those alone. And then not acknowledge how the massive group of participants that you _chose_ to leave out could have affected the results, or how the choice to leave them out affects the null hypothesis. Hope this helps! (I'm not a researcher but I work in academia and have on occasion assisted in experimental design.)