3 ms·

> In fact, the Covid stuff was added after the study had started, Not relevant to my point. They didn't verify infection. The people being surveyed knew that t

by timr 5y ago

> In fact, the Covid stuff was added after the study had started, Not relevant to my point. They didn't verify infection. The people being surveyed knew that they were being surveyed about Covid-related things. Also, even if you separate the intelligence-test part from the covid-results part by time, now you have two problems: self selection bias for the intelligence part (i.e. people who are more likely to be worried about cognitive decline take the time to participate in tests of cognitive skills), and for the covid test part (i.e. people who are worried about Covid will be more likely to respond to the Covid test part). There are so many sources of bias here, it's impossible to de-combobulate them all. It's simply a mess. > and they did a follow-up survey in December 2020 and of the people who reported later Covid, they were indistinguishable from the overall population. You can attempt to control for bias by doing this sort of thing, but it's not enough. Just because the sample population looked like the control population on summary parameters B, C and Z doesn't mean that they weren't biased on the primary endpoint of the study -- and remember. the participants knew they were being surveyed related to Covid (because again, they were being asked). Also, the procedure they used also leaves open the possibility that (for example) study participants could have self-selected by finding the survey from "long covid" forums, and the like. We can't rule it out. > Is your argument that people would preferentially report Covid-19 and throw the intelligence test? Or something else? I have a number of problems with this study (and I've enumerated a few) but most of them fall under the category of "self-reported data is garbage", which is why I led with that. As the French CONSTANCES study (a much better study) I linked showed: when people are allowed to self-report even just their covid status, they mess up the results. A better (but still not good) study of this sort would at least take the intelligence test and cross-check it against testing results to get an objective result on one of the parameters of interest. But there would still be too many sources of bias to enumerate -- open-ended, web-based surveys like "The Great British Intelligence Test" are simply garbage, and should not be taken seriously by anyone, anywhere.

disgruntledphd2 5y ago
OK thank you for expressing your issues with the paper in a much more useful fashion that your first take, it's greatly appreciated. I would note, however, that your strong point that self-reported data is garbage would lead you to disbelieve in the existence of pain, which seems like an unlikely interpretation. That French study is interesting, I need to dig into it a little more. That being said, do you think that people lied about being in hospital/on ventilator. That seems unlikely to me, and the cognitive outcomes seem to respond in a dose-dependent fashion to each step of Covid infection, which is not something that I would have expected to see if there were response biases going on. Like, the only real explanation I can come up with here is that maybe a bunch of people were still sick, and this correlation drives the observed responses with cognitive function. Looking at the methods, the Covid stuff comes after the cognitive test, so could not have impacted the cognitive test results. Additionally the actual webform (linked from the paper) shows nothing about Covid. Again, I think the biggest bias in this sample is likely to be their TV-watching status. That being said, they did a complete cases analysis, which can often throw away lots of useful data, but it doesn't appear that many people quit halfway through (which is weird, given my experience of doing survey research). > As the French CONSTANCES study (a much better study) I linked showed: when people are allowed to self-report even just their covid status, they mess up the results. This comes under the headings of different disciplines are different I guess. If this study had been done under the aegis of a medical department, then we'd have laboratory confirmed Covid but they'd have messed up the cognitive test part. Science is difficult, and messy and often not satisfying to anyone, and personally I think we can learn things from this paper. You disagree, I guess we'll find out when the systematic reviews/meta-analyses happen ;)
- timr 5y ago
  > That being said, do you think that people lied about being in hospital/on ventilator. That seems unlikely to me, and the cognitive outcomes seem to respond in a dose-dependent fashion to each step of Covid infection, which is not something that I would have expected to see if there were response biases going on. I said in my original comment that we need to be more careful about separating the degrees of severity of illness from reports of "long covid". Post-hospitalization syndrome is a thing -- people who were on ventilators have lingering after effects. It isn't "long covid", and confusing that with reports of "I had asymptomatic illness and now I am tired" is just bad science. Unfortunately, most of these "long covid" studies do exactly that. They take all levels of severity, mix them together, correlate with self-reports of symptoms, and don't anchor anything to objectively observable criteria (like test results). Most of this stuff wouldn't pass muster at an 8th grade science fair. > Looking at the methods, the Covid stuff comes after the cognitive test, so could not have impacted the cognitive test results. You have no evidence that they had covid at the time of the cognitive test (or before). It's self-reported. Even if it's true that the two questions were strictly separated in time, and it wasn't possible to take the cognitive test later, after the covid questions were added (this is not clear from the methods, btw), the results are hopelessly confounded. > Again, I think the biggest bias in this sample is likely to be their TV-watching status. They recruited across the web as well. It's right there in the methods.

by timr 5y ago

disgruntledphd2 5y ago

OK thank you for expressing your issues with the paper in a much more useful fashion that your first take, it's greatly appreciated. I would note, however, that your strong point that self-reported data is garbage would lead you to disbelieve in the existence of pain, which seems like an unlikely interpretation. That French study is interesting, I need to dig into it a little more. That being said, do you think that people lied about being in hospital/on ventilator. That seems unlikely to me, and the cognitive outcomes seem to respond in a dose-dependent fashion to each step of Covid infection, which is not something that I would have expected to see if there were response biases going on. Like, the only real explanation I can come up with here is that maybe a bunch of people were still sick, and this correlation drives the observed responses with cognitive function. Looking at the methods, the Covid stuff comes after the cognitive test, so could not have impacted the cognitive test results. Additionally the actual webform (linked from the paper) shows nothing about Covid. Again, I think the biggest bias in this sample is likely to be their TV-watching status. That being said, they did a complete cases analysis, which can often throw away lots of useful data, but it doesn't appear that many people quit halfway through (which is weird, given my experience of doing survey research). > As the French CONSTANCES study (a much better study) I linked showed: when people are allowed to self-report even just their covid status, they mess up the results. This comes under the headings of different disciplines are different I guess. If this study had been done under the aegis of a medical department, then we'd have laboratory confirmed Covid but they'd have messed up the cognitive test part. Science is difficult, and messy and often not satisfying to anyone, and personally I think we can learn things from this paper. You disagree, I guess we'll find out when the systematic reviews/meta-analyses happen ;)

timr 5y ago
> That being said, do you think that people lied about being in hospital/on ventilator. That seems unlikely to me, and the cognitive outcomes seem to respond in a dose-dependent fashion to each step of Covid infection, which is not something that I would have expected to see if there were response biases going on. I said in my original comment that we need to be more careful about separating the degrees of severity of illness from reports of "long covid". Post-hospitalization syndrome is a thing -- people who were on ventilators have lingering after effects. It isn't "long covid", and confusing that with reports of "I had asymptomatic illness and now I am tired" is just bad science. Unfortunately, most of these "long covid" studies do exactly that. They take all levels of severity, mix them together, correlate with self-reports of symptoms, and don't anchor anything to objectively observable criteria (like test results). Most of this stuff wouldn't pass muster at an 8th grade science fair. > Looking at the methods, the Covid stuff comes after the cognitive test, so could not have impacted the cognitive test results. You have no evidence that they had covid at the time of the cognitive test (or before). It's self-reported. Even if it's true that the two questions were strictly separated in time, and it wasn't possible to take the cognitive test later, after the covid questions were added (this is not clear from the methods, btw), the results are hopelessly confounded. > Again, I think the biggest bias in this sample is likely to be their TV-watching status. They recruited across the web as well. It's right there in the methods.