4 ms·

The data was from a cognitive assessment, not self-reported. There were some self-reported components, like do you have depression, anxiety, etc. But the measur

The data was from a cognitive assessment, not self-reported. There were some self-reported components, like do you have depression, anxiety, etc. But the measurement of cognition was not self reported.

timr 5y ago
The question of whether or not the respondent actually had Covid-19 was self-reported: > People who indicated that they had suspected having COVID-19 were presented further questions including whether they had breathing difficulties, what happened as a consequence of their breathing difficulties, and whether there had been positive confirmation via a biological test (Table S1). Also, this was a web-based test, conducted at home, where the participants self-selected into the survey. There is no universe in which this should be considered unbiased data.
- halpert 5y ago
  Sure, the data isn't perfect, but data rarely is. You're also assuming that, because the data isn't perfect, that the effect measured would go away with better data. I actually think the opposite would happen. There is a correlation between the severity of self-reported sickness and cognitive decline. Those who ended up in a hospital show the most severe signs of cognitive decline. If they were actually in a hospital, they likely know for sure whether or not they were positive, compared to those who recovered at home and only suspected they had covid. There have also been visible changes to the brain when looking at MRIs before / after a person has covid.
- disgruntledphd2 5y ago
  > Also, this was a web-based test, conducted at home, where the participants self-selected into the survey. There is no universe in which this should be considered unbiased data. You know, I was kinda in agreement with you until you said this. Please read the paper (which had absolutely no information about the Covid stuff in the docs or signup forms.) Like, one could argue that the major selection factor was whether or not the subject watched TV, which would be fair but not germane to this particular argument. In fact, the Covid stuff was added after the study had started, and they did a follow-up survey in December 2020 and of the people who reported later Covid, they were indistinguishable from the overall population. Like, I agree that this would have been much much better if they'd had access to the NHS data and could have joined with the cognitive test data and I personally am not convinced of their modelling approaches, but there's a tradeoff between enough data to estimate important effects, and the accuracy of said data, especially when it comes to medical conditions. Is your argument that people would preferentially report Covid-19 and throw the intelligence test? Or something else? Like, this paper has a bunch of problems, but the main effect is not that surprising. The real question is how generalisable the findings are to other viral infections. That's something that hopefully will get investigated more over time.
  - timr 5y ago
    > In fact, the Covid stuff was added after the study had started, Not relevant to my point. They didn't verify infection. The people being surveyed knew that they were being surveyed about Covid-related things. Also, even if you separate the intelligence-test part from the covid-results part by time, now you have two problems: self selection bias for the intelligence part (i.e. people who are more likely to be worried about cognitive decline take the time to participate in tests of cognitive skills), and for the covid test part (i.e. people who are worried about Covid will be more likely to respond to the Covid test part). There are so many sources of bias here, it's impossible to de-combobulate them all. It's simply a mess. > and they did a follow-up survey in December 2020 and of the people who reported later Covid, they were indistinguishable from the overall population. You can attempt to control for bias by doing this sort of thing, but it's not enough. Just because the sample population looked like the control population on summary parameters B, C and Z doesn't mean that they weren't biased on the primary endpoint of the study -- and remember. the participants knew they were being surveyed related to Covid (because again, they were being asked). Also, the procedure they used also leaves open the possibility that (for example) study participants could have self-selected by finding the survey from "long covid" forums, and the like. We can't rule it out. > Is your argument that people would preferentially report Covid-19 and throw the intelligence test? Or something else? I have a number of problems with this study (and I've enumerated a few) but most of them fall under the category of "self-reported data is garbage", which is why I led with that. As the French CONSTANCES study (a much better study) I linked showed: when people are allowed to self-report even just their covid status, they mess up the results. A better (but still not good) study of this sort would at least take the intelligence test and cross-check it against testing results to get an objective result on one of the parameters of interest. But there would still be too many sources of bias to enumerate -- open-ended, web-based surveys like "The Great British Intelligence Test" are simply garbage, and should not be taken seriously by anyone, anywhere.
    - disgruntledphd2 5y ago
      OK thank you for expressing your issues with the paper in a much more useful fashion that your first take, it's greatly appreciated. I would note, however, that your strong point that self-reported data is garbage would lead you to disbelieve in the existence of pain, which seems like an unlikely interpretation. That French study is interesting, I need to dig into it a little more. That being said, do you think that people lied about being in hospital/on ventilator. That seems unlikely to me, and the cognitive outcomes seem to respond in a dose-dependent fashion to each step of Covid infection, which is not something that I would have expected to see if there were response biases going on. Like, the only real explanation I can come up with here is that maybe a bunch of people were still sick, and this correlation drives the observed responses with cognitive function. Looking at the methods, the Covid stuff comes after the cognitive test, so could not have impacted the cognitive test results. Additionally the actual webform (linked from the paper) shows nothing about Covid. Again, I think the biggest bias in this sample is likely to be their TV-watching status. That being said, they did a complete cases analysis, which can often throw away lots of useful data, but it doesn't appear that many people quit halfway through (which is weird, given my experience of doing survey research). > As the French CONSTANCES study (a much better study) I linked showed: when people are allowed to self-report even just their covid status, they mess up the results. This comes under the headings of different disciplines are different I guess. If this study had been done under the aegis of a medical department, then we'd have laboratory confirmed Covid but they'd have messed up the cognitive test part. Science is difficult, and messy and often not satisfying to anyone, and personally I think we can learn things from this paper. You disagree, I guess we'll find out when the systematic reviews/meta-analyses happen ;)
deleted 5y ago
[deleted]

timr 5y ago

The question of whether or not the respondent actually had Covid-19 was self-reported: > People who indicated that they had suspected having COVID-19 were presented further questions including whether they had breathing difficulties, what happened as a consequence of their breathing difficulties, and whether there had been positive confirmation via a biological test (Table S1). Also, this was a web-based test, conducted at home, where the participants self-selected into the survey. There is no universe in which this should be considered unbiased data.

halpert 5y ago
Sure, the data isn't perfect, but data rarely is. You're also assuming that, because the data isn't perfect, that the effect measured would go away with better data. I actually think the opposite would happen. There is a correlation between the severity of self-reported sickness and cognitive decline. Those who ended up in a hospital show the most severe signs of cognitive decline. If they were actually in a hospital, they likely know for sure whether or not they were positive, compared to those who recovered at home and only suspected they had covid. There have also been visible changes to the brain when looking at MRIs before / after a person has covid.
disgruntledphd2 5y ago
> Also, this was a web-based test, conducted at home, where the participants self-selected into the survey. There is no universe in which this should be considered unbiased data. You know, I was kinda in agreement with you until you said this. Please read the paper (which had absolutely no information about the Covid stuff in the docs or signup forms.) Like, one could argue that the major selection factor was whether or not the subject watched TV, which would be fair but not germane to this particular argument. In fact, the Covid stuff was added after the study had started, and they did a follow-up survey in December 2020 and of the people who reported later Covid, they were indistinguishable from the overall population. Like, I agree that this would have been much much better if they'd had access to the NHS data and could have joined with the cognitive test data and I personally am not convinced of their modelling approaches, but there's a tradeoff between enough data to estimate important effects, and the accuracy of said data, especially when it comes to medical conditions. Is your argument that people would preferentially report Covid-19 and throw the intelligence test? Or something else? Like, this paper has a bunch of problems, but the main effect is not that surprising. The real question is how generalisable the findings are to other viral infections. That's something that hopefully will get investigated more over time.
- timr 5y ago
  > In fact, the Covid stuff was added after the study had started, Not relevant to my point. They didn't verify infection. The people being surveyed knew that they were being surveyed about Covid-related things. Also, even if you separate the intelligence-test part from the covid-results part by time, now you have two problems: self selection bias for the intelligence part (i.e. people who are more likely to be worried about cognitive decline take the time to participate in tests of cognitive skills), and for the covid test part (i.e. people who are worried about Covid will be more likely to respond to the Covid test part). There are so many sources of bias here, it's impossible to de-combobulate them all. It's simply a mess. > and they did a follow-up survey in December 2020 and of the people who reported later Covid, they were indistinguishable from the overall population. You can attempt to control for bias by doing this sort of thing, but it's not enough. Just because the sample population looked like the control population on summary parameters B, C and Z doesn't mean that they weren't biased on the primary endpoint of the study -- and remember. the participants knew they were being surveyed related to Covid (because again, they were being asked). Also, the procedure they used also leaves open the possibility that (for example) study participants could have self-selected by finding the survey from "long covid" forums, and the like. We can't rule it out. > Is your argument that people would preferentially report Covid-19 and throw the intelligence test? Or something else? I have a number of problems with this study (and I've enumerated a few) but most of them fall under the category of "self-reported data is garbage", which is why I led with that. As the French CONSTANCES study (a much better study) I linked showed: when people are allowed to self-report even just their covid status, they mess up the results. A better (but still not good) study of this sort would at least take the intelligence test and cross-check it against testing results to get an objective result on one of the parameters of interest. But there would still be too many sources of bias to enumerate -- open-ended, web-based surveys like "The Great British Intelligence Test" are simply garbage, and should not be taken seriously by anyone, anywhere.
  - disgruntledphd2 5y ago
    OK thank you for expressing your issues with the paper in a much more useful fashion that your first take, it's greatly appreciated. I would note, however, that your strong point that self-reported data is garbage would lead you to disbelieve in the existence of pain, which seems like an unlikely interpretation. That French study is interesting, I need to dig into it a little more. That being said, do you think that people lied about being in hospital/on ventilator. That seems unlikely to me, and the cognitive outcomes seem to respond in a dose-dependent fashion to each step of Covid infection, which is not something that I would have expected to see if there were response biases going on. Like, the only real explanation I can come up with here is that maybe a bunch of people were still sick, and this correlation drives the observed responses with cognitive function. Looking at the methods, the Covid stuff comes after the cognitive test, so could not have impacted the cognitive test results. Additionally the actual webform (linked from the paper) shows nothing about Covid. Again, I think the biggest bias in this sample is likely to be their TV-watching status. That being said, they did a complete cases analysis, which can often throw away lots of useful data, but it doesn't appear that many people quit halfway through (which is weird, given my experience of doing survey research). > As the French CONSTANCES study (a much better study) I linked showed: when people are allowed to self-report even just their covid status, they mess up the results. This comes under the headings of different disciplines are different I guess. If this study had been done under the aegis of a medical department, then we'd have laboratory confirmed Covid but they'd have messed up the cognitive test part. Science is difficult, and messy and often not satisfying to anyone, and personally I think we can learn things from this paper. You disagree, I guess we'll find out when the systematic reviews/meta-analyses happen ;)

deleted 5y ago

[deleted]