4 ms·

If the noise in polygenic scores is random, all that will do is reduce correlations. Random measurement error always reduces the ability to observe relationship

If the noise in polygenic scores is random, all that will do is reduce correlations. Random measurement error always reduces the ability to observe relationships. To be clear, I do not know enough about polygenic scores to judge one way or another how the noise "works" vis-a-vis the entire analysis. The Porsche comment is snide, but actually exposes a similar error in your critique. Sure, a tax return-derived measure of income would be superior to measuring if someone owned a luxury car. But, if you found yourself in a situation where all you had to go on for measuring economic wellbeing was (luxury) car ownership, your analysis is likely to improve by including it rather than excluding it, unless the measure itself had serious other issues with its accuracy. Likewise, for SES, it is an imperfect measure, but it is the best we have for measuring social position in a concise way. Having worked in research and universities for a while, the type of critique presented in this post is one you often see of new graduate students. They are able to tear down problems with research very well, but tend to overlook whether the study itself was still informative, or whether the opposite finding is likely to be true. For example, suppose we wanted to know if video games or watching videos on the internet are making you dumber. A study like this may not convince you it's making you smarter, but it presents decent evidence they're not making you dumber. You can point out how the measures aren't perfect, but that is far from saying the opposite is true or the observed trends are completely spurious.

dash2 4y ago
The point is not whether including polygenic scores is better than nothing. The point is whether it's good enough to justify the claims they are making. It's not. The same holds for SES. I disagree that this study presents decent evidence of anything. I don't claim that the conclusions are false. But they haven't backed them up. There are lots of ways that the observed trends can be spurious. I mentioned some. The study is very weakly informative.
- DANK_YACHT 4y ago
  > The point is not whether including polygenic scores is better than nothing. The point is whether it's good enough to justify the claims they are making. It's not. What is the justification for this assertion? If polygenic scores are simply "noisy," then, as the GP mentioned, they may be good enough when used in aggregate. There can be a lot of signal in noisy data. Ask any ML practitioner.
  - dash2 4y ago
    Polygenic scores are themselves an aggregation. And they contain a ton of noise. From twin studies, R2 of the "true" polygenic score on education is about 40%. The R2 of our best actual polygenic score is 4%. IIRC these guys aren't even using the education polygenic score. They're using the PGS for IQ. That's even noisier (because it was created using smaller samples, and you need biiig samples for this to work).
    - DANK_YACHT 4y ago
      > Polygenic scores are themselves an aggregation. And they contain a ton of noise. Again, noisy data can still be useful. For instance, generate a perfect single-variable normal distribution, sample along the x-axis and perturb each point randomly in the y direction either up or down. Depending on the range of random values used to move the sampled points, you can still see the underlying distribution even though the data is noisy. Two possible arguments you might make: 1) The data is noisy and the paper's authors haven't collected enough data to account for the amount of noise. Usually people will do something like null-hypothesis significance testing to measure this. 2) The noise isn't uniformly random and has some underlying bias that is affecting the results.
      - dash2 4y ago
        Noisy data can be useful. What it can't do is be useful enough to say "we have controlled for X". There will be a large unmeasured component of genetic variation - probably about 90% of it - that controlling for the PGS doesn't capture. Given that, throwing a PGS into the regression is not worth the candle.
6gvONxR4sf7o 4y ago
> If the noise in polygenic scores is random, all that will do is reduce correlations. Random measurement error always reduces the ability to observe relationships. I think you’re misunderstanding how they’re being used (or I am). I think they’re trying to control for genetics via polygenic scores, not trying to establish a relationship between those scores and intelligence. The analogy is that you’re measuring the effect of the price of kids’ socks on their intelligence, and saying the observed effect isn’t due to parental income in some other way, because you’ve controlled for parental income(by controlling for whether there’s a porche in the driveway).
gwern 4y ago
> If the noise in polygenic scores is random, all that will do is reduce correlations. Random measurement error always reduces the ability to observe relationships. To be clear, I do not know enough about polygenic scores to judge one way or another how the noise "works" vis-a-vis the entire analysis. That's correct, it is a flaw of the entire analysis, not the PGS in isolation. Yes, the polygenic score, when used to define 'genetic intelligence', will be biased towards zero and will miss a lot of the genetic intelligence. What then happens is the video-game playing becomes a measure of intelligence (genetic or otherwise), capturing what the polygenic score (and other covariates) miss. The logic then works in reverse: the reduced correlation is precisely why the residual confounding works. The worse your 'measurements' are at measuring the underlying trait, the more wiggle room there is for your 'outcomes' to actually be correcting the 'measurements' and not vice versa. See "Statistically Controlling for Confounding Constructs Is Harder than You Think" https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152719 https://journals.plos.org/plosone/article?id=10.1371/journal... , Westfall & Yarkoni 2016. (More examples: https://www.gwern.net/notes/Regression https://www.gwern.net/notes/Regression ) What OP shows is not that video game playing causes IQ, but IQ causes video game playing. The choice to play video games (or not play them, because you are bad at learning) is an additional 1-item long IQ test and helps corrects for the error. (And we do in fact know that video gaming & IQ correlate, so nothing new there. We also know from all the brain training randomized experiments that the causal arrow doesn't run in the direction they want it to run. OP is very wrong, including in claiming that the Flynn effect justifies believing in their effect - it actually is a criticism of their claimed causal relationship between IQs have been steady or falling even as video gaming increased massively.)

dash2 4y ago

The point is not whether including polygenic scores is better than nothing. The point is whether it's good enough to justify the claims they are making. It's not. The same holds for SES. I disagree that this study presents decent evidence of anything. I don't claim that the conclusions are false. But they haven't backed them up. There are lots of ways that the observed trends can be spurious. I mentioned some. The study is very weakly informative.

DANK_YACHT 4y ago
> The point is not whether including polygenic scores is better than nothing. The point is whether it's good enough to justify the claims they are making. It's not. What is the justification for this assertion? If polygenic scores are simply "noisy," then, as the GP mentioned, they may be good enough when used in aggregate. There can be a lot of signal in noisy data. Ask any ML practitioner.
- dash2 4y ago
  Polygenic scores are themselves an aggregation. And they contain a ton of noise. From twin studies, R2 of the "true" polygenic score on education is about 40%. The R2 of our best actual polygenic score is 4%. IIRC these guys aren't even using the education polygenic score. They're using the PGS for IQ. That's even noisier (because it was created using smaller samples, and you need biiig samples for this to work).
  - DANK_YACHT 4y ago
    > Polygenic scores are themselves an aggregation. And they contain a ton of noise. Again, noisy data can still be useful. For instance, generate a perfect single-variable normal distribution, sample along the x-axis and perturb each point randomly in the y direction either up or down. Depending on the range of random values used to move the sampled points, you can still see the underlying distribution even though the data is noisy. Two possible arguments you might make: 1) The data is noisy and the paper's authors haven't collected enough data to account for the amount of noise. Usually people will do something like null-hypothesis significance testing to measure this. 2) The noise isn't uniformly random and has some underlying bias that is affecting the results.
    - dash2 4y ago
      Noisy data can be useful. What it can't do is be useful enough to say "we have controlled for X". There will be a large unmeasured component of genetic variation - probably about 90% of it - that controlling for the PGS doesn't capture. Given that, throwing a PGS into the regression is not worth the candle.

6gvONxR4sf7o 4y ago

> If the noise in polygenic scores is random, all that will do is reduce correlations. Random measurement error always reduces the ability to observe relationships. I think you’re misunderstanding how they’re being used (or I am). I think they’re trying to control for genetics via polygenic scores, not trying to establish a relationship between those scores and intelligence. The analogy is that you’re measuring the effect of the price of kids’ socks on their intelligence, and saying the observed effect isn’t due to parental income in some other way, because you’ve controlled for parental income(by controlling for whether there’s a porche in the driveway).

gwern 4y ago

> If the noise in polygenic scores is random, all that will do is reduce correlations. Random measurement error always reduces the ability to observe relationships. To be clear, I do not know enough about polygenic scores to judge one way or another how the noise "works" vis-a-vis the entire analysis. That's correct, it is a flaw of the entire analysis, not the PGS in isolation. Yes, the polygenic score, when used to define 'genetic intelligence', will be biased towards zero and will miss a lot of the genetic intelligence. What then happens is the video-game playing becomes a measure of intelligence (genetic or otherwise), capturing what the polygenic score (and other covariates) miss. The logic then works in reverse: the reduced correlation is precisely why the residual confounding works. The worse your 'measurements' are at measuring the underlying trait, the more wiggle room there is for your 'outcomes' to actually be correcting the 'measurements' and not vice versa. See "Statistically Controlling for Confounding Constructs Is Harder than You Think" https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152719 https://journals.plos.org/plosone/article?id=10.1371/journal... , Westfall & Yarkoni 2016. (More examples: https://www.gwern.net/notes/Regression https://www.gwern.net/notes/Regression ) What OP shows is not that video game playing causes IQ, but IQ causes video game playing. The choice to play video games (or not play them, because you are bad at learning) is an additional 1-item long IQ test and helps corrects for the error. (And we do in fact know that video gaming & IQ correlate, so nothing new there. We also know from all the brain training randomized experiments that the causal arrow doesn't run in the direction they want it to run. OP is very wrong, including in claiming that the Flynn effect justifies believing in their effect - it actually is a criticism of their claimed causal relationship between IQs have been steady or falling even as video gaming increased massively.)