4 ms·

This is self-reported data. Self-reported data is garbage. There's simply too much risk of bias. To illustrate the point, a large study out of France found tha

by timr 5y ago

This is self-reported data. Self-reported data is garbage. There's simply too much risk of bias. To illustrate the point, a large study out of France found that the only "long covid" symptom that actually correlates with confirmed illness was loss of smell. But there were multiple "long covid" symptoms that correlated with self-reported illness. https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2785832 https://jamanetwork.com/journals/jamainternalmedicine/fullar... We have to be careful to separate populations by severity of disease, hospitalization, etc., but "long covid" is at least partly psychosomatic.

halpert 5y ago
The data was from a cognitive assessment, not self-reported. There were some self-reported components, like do you have depression, anxiety, etc. But the measurement of cognition was not self reported.
- timr 5y ago
  The question of whether or not the respondent actually had Covid-19 was self-reported: > People who indicated that they had suspected having COVID-19 were presented further questions including whether they had breathing difficulties, what happened as a consequence of their breathing difficulties, and whether there had been positive confirmation via a biological test (Table S1). Also, this was a web-based test, conducted at home, where the participants self-selected into the survey. There is no universe in which this should be considered unbiased data.
  - halpert 5y ago
    Sure, the data isn't perfect, but data rarely is. You're also assuming that, because the data isn't perfect, that the effect measured would go away with better data. I actually think the opposite would happen. There is a correlation between the severity of self-reported sickness and cognitive decline. Those who ended up in a hospital show the most severe signs of cognitive decline. If they were actually in a hospital, they likely know for sure whether or not they were positive, compared to those who recovered at home and only suspected they had covid. There have also been visible changes to the brain when looking at MRIs before / after a person has covid.
  - disgruntledphd2 5y ago
    > Also, this was a web-based test, conducted at home, where the participants self-selected into the survey. There is no universe in which this should be considered unbiased data. You know, I was kinda in agreement with you until you said this. Please read the paper (which had absolutely no information about the Covid stuff in the docs or signup forms.) Like, one could argue that the major selection factor was whether or not the subject watched TV, which would be fair but not germane to this particular argument. In fact, the Covid stuff was added after the study had started, and they did a follow-up survey in December 2020 and of the people who reported later Covid, they were indistinguishable from the overall population. Like, I agree that this would have been much much better if they'd had access to the NHS data and could have joined with the cognitive test data and I personally am not convinced of their modelling approaches, but there's a tradeoff between enough data to estimate important effects, and the accuracy of said data, especially when it comes to medical conditions. Is your argument that people would preferentially report Covid-19 and throw the intelligence test? Or something else? Like, this paper has a bunch of problems, but the main effect is not that surprising. The real question is how generalisable the findings are to other viral infections. That's something that hopefully will get investigated more over time.
    - timr 5y ago
      > In fact, the Covid stuff was added after the study had started, Not relevant to my point. They didn't verify infection. The people being surveyed knew that they were being surveyed about Covid-related things. Also, even if you separate the intelligence-test part from the covid-results part by time, now you have two problems: self selection bias for the intelligence part (i.e. people who are more likely to be worried about cognitive decline take the time to participate in tests of cognitive skills), and for the covid test part (i.e. people who are worried about Covid will be more likely to respond to the Covid test part). There are so many sources of bias here, it's impossible to de-combobulate them all. It's simply a mess. > and they did a follow-up survey in December 2020 and of the people who reported later Covid, they were indistinguishable from the overall population. You can attempt to control for bias by doing this sort of thing, but it's not enough. Just because the sample population looked like the control population on summary parameters B, C and Z doesn't mean that they weren't biased on the primary endpoint of the study -- and remember. the participants knew they were being surveyed related to Covid (because again, they were being asked). Also, the procedure they used also leaves open the possibility that (for example) study participants could have self-selected by finding the survey from "long covid" forums, and the like. We can't rule it out. > Is your argument that people would preferentially report Covid-19 and throw the intelligence test? Or something else? I have a number of problems with this study (and I've enumerated a few) but most of them fall under the category of "self-reported data is garbage", which is why I led with that. As the French CONSTANCES study (a much better study) I linked showed: when people are allowed to self-report even just their covid status, they mess up the results. A better (but still not good) study of this sort would at least take the intelligence test and cross-check it against testing results to get an objective result on one of the parameters of interest. But there would still be too many sources of bias to enumerate -- open-ended, web-based surveys like "The Great British Intelligence Test" are simply garbage, and should not be taken seriously by anyone, anywhere.
- deleted 5y ago
  [deleted]

by timr 5y ago

halpert 5y ago

The data was from a cognitive assessment, not self-reported. There were some self-reported components, like do you have depression, anxiety, etc. But the measurement of cognition was not self reported.

timr 5y ago
The question of whether or not the respondent actually had Covid-19 was self-reported: > People who indicated that they had suspected having COVID-19 were presented further questions including whether they had breathing difficulties, what happened as a consequence of their breathing difficulties, and whether there had been positive confirmation via a biological test (Table S1). Also, this was a web-based test, conducted at home, where the participants self-selected into the survey. There is no universe in which this should be considered unbiased data.
- halpert 5y ago
  Sure, the data isn't perfect, but data rarely is. You're also assuming that, because the data isn't perfect, that the effect measured would go away with better data. I actually think the opposite would happen. There is a correlation between the severity of self-reported sickness and cognitive decline. Those who ended up in a hospital show the most severe signs of cognitive decline. If they were actually in a hospital, they likely know for sure whether or not they were positive, compared to those who recovered at home and only suspected they had covid. There have also been visible changes to the brain when looking at MRIs before / after a person has covid.
- disgruntledphd2 5y ago
  > Also, this was a web-based test, conducted at home, where the participants self-selected into the survey. There is no universe in which this should be considered unbiased data. You know, I was kinda in agreement with you until you said this. Please read the paper (which had absolutely no information about the Covid stuff in the docs or signup forms.) Like, one could argue that the major selection factor was whether or not the subject watched TV, which would be fair but not germane to this particular argument. In fact, the Covid stuff was added after the study had started, and they did a follow-up survey in December 2020 and of the people who reported later Covid, they were indistinguishable from the overall population. Like, I agree that this would have been much much better if they'd had access to the NHS data and could have joined with the cognitive test data and I personally am not convinced of their modelling approaches, but there's a tradeoff between enough data to estimate important effects, and the accuracy of said data, especially when it comes to medical conditions. Is your argument that people would preferentially report Covid-19 and throw the intelligence test? Or something else? Like, this paper has a bunch of problems, but the main effect is not that surprising. The real question is how generalisable the findings are to other viral infections. That's something that hopefully will get investigated more over time.
  - timr 5y ago
    > In fact, the Covid stuff was added after the study had started, Not relevant to my point. They didn't verify infection. The people being surveyed knew that they were being surveyed about Covid-related things. Also, even if you separate the intelligence-test part from the covid-results part by time, now you have two problems: self selection bias for the intelligence part (i.e. people who are more likely to be worried about cognitive decline take the time to participate in tests of cognitive skills), and for the covid test part (i.e. people who are worried about Covid will be more likely to respond to the Covid test part). There are so many sources of bias here, it's impossible to de-combobulate them all. It's simply a mess. > and they did a follow-up survey in December 2020 and of the people who reported later Covid, they were indistinguishable from the overall population. You can attempt to control for bias by doing this sort of thing, but it's not enough. Just because the sample population looked like the control population on summary parameters B, C and Z doesn't mean that they weren't biased on the primary endpoint of the study -- and remember. the participants knew they were being surveyed related to Covid (because again, they were being asked). Also, the procedure they used also leaves open the possibility that (for example) study participants could have self-selected by finding the survey from "long covid" forums, and the like. We can't rule it out. > Is your argument that people would preferentially report Covid-19 and throw the intelligence test? Or something else? I have a number of problems with this study (and I've enumerated a few) but most of them fall under the category of "self-reported data is garbage", which is why I led with that. As the French CONSTANCES study (a much better study) I linked showed: when people are allowed to self-report even just their covid status, they mess up the results. A better (but still not good) study of this sort would at least take the intelligence test and cross-check it against testing results to get an objective result on one of the parameters of interest. But there would still be too many sources of bias to enumerate -- open-ended, web-based surveys like "The Great British Intelligence Test" are simply garbage, and should not be taken seriously by anyone, anywhere.
deleted 5y ago
[deleted]