10 ms·

The fact that they "controlled" for genetics using polygenic scores already is a strong sign of low quality research. Polygenic scores are powerful, but they co

by dash2 4y ago

The fact that they "controlled" for genetics using polygenic scores already is a strong sign of low quality research. Polygenic scores are powerful, but they contain very large amounts of noise compared to the true genetic effects. Controlling for genetics using them is like controlling for income by asking whether the respondents own a Porsche. Also, be aware that Scientific Reports is, if not quite a predatory journal, a very low bar. They publish tens of thousands of articles every year, while charging vast fees. In general, these guys have correlations, not causation. Children's IQ - and gaming habits etc. - develop as they age, so controlling for baseline IQ is not enough to make a correlation with later IQ and gaming causal. It seems much more likely that smarter kids game more, e.g. because they live in richer households. (No, controlling for SES isn't enough to rule this out, for much the same reasons of measurement error as for the genetics.) If you wanna believe that your hours on COD have made you a genius, go ahead, I won't stop you. Just don't imagine that this research proves it.

nicce 4y ago
The cost is no joke. ”Scientific Reports is an open access journal. To publish in Scientific Reports, all authors are required to pay an article-processing charge (APC) of $1,495.”
- napier 4y ago
  “Open access” journal fees are often accounted for by institutional affiliations or grant funding donors. The upside of upfront publication charges is that the research output isn’t paywalled and therefore ideally reaches more eyeballs and brainpans than the conventional route.
  - nicce 4y ago
    That is the upside, but there are many other high reputation peer-reviewed open journals with close to zero payment. Is the domain really worth the cost? It is kinda messed up, when you adjust the price based on the available money of institutional affiliations or donors. It is just more profits for journal and nothing for authors, while forcing authors to pay.
    - Aeolun 4y ago
      > there are many other high reputation peer-reviewed open journals with close to zero payment Which are those? The journals that I expect to still be up are the ones that require actual payment for publishing.
      - nicce 4y ago
        For example accepted papers in HICSS are stored publicly without extra cost. > Starting with HICSS-50, all HICSS publication will be archived and disseminated at no cost to all readers worldwide through ScholarSpace. A complete set of HICSS proceedings will also be made available in the digital library of the Association of Information Systems (AIS). https://hicss.hawaii.edu/resources/ https://hicss.hawaii.edu/resources/
  - svnt 4y ago
    Most grants in the fields I’ve looked at preclude their use for paying publishing fees.
- bigmadshoe 4y ago
  This is becoming very common in academia. The university I used to work for is beginning to require that all research is published in open access journals. Do you know the ludicrous fees universities pay to closed-access journals?
- BeetleB 4y ago
  This is not an unusual charge for open access journals. The New Journal of Physics, a respected open access physics journal, charges $2225. https://publishingsupport.iopscience.iop.org/journals/new-journal-of-physics/about-new-journal-physics/#publication-charges https://publishingsupport.iopscience.iop.org/journals/new-jo...
throwingrocks 4y ago
Your response seemed reasonable, and I was nodding along until the last paragraph. I get the impression you had your mind made up about video games and intelligence long before this study was published.
- lettergram 4y ago
  I think the point the prior commenter was making was that this research validates people who think gaming is good. They pointed out the issues and said “if you still want to believe this, I can’t stop you” The commenter may have a bias, but most prior research shows us the opposite of the study. I also saw the poor experimental design and had a similar thoughts. Basically, this research looks poorly done and like an effort to prop up the gaming industry (and / or validate the authors pre-suppositions).
  - d3ckard 4y ago
    In any case, using CoD as an example does not exactly show good will. I personally believe in huge gains from gaming, based on personal experience (so obviously n = 1, read further accordingly). Platformers train hand-eye coordination and pattern recognition, strategies teach resource management, RPGs about optimization and adopting growth mindset, racing games require long-time concentration, puzzlers and adventure games test your logic. In general, games require you to: - learn a set of tools - master them - conquer objective while also prevailing in face of adversity. I never regretted the hours I spent gaming and I feel they contributed very much to my softeng career (not directly though).
    - guerrilla 4y ago
      > good will. I think you meant good faith. They didn't use it as an example though, it was just rhetorical.
    - hackernewds 4y ago
      The question is about incrementality. in those same hours you played CoD, do you think (assuming the endpoint as you suggest is being a softeng) you could have learned more by practicing softeng (even if for half the time)
versteegen 4y ago
Note though that their data comes from the ABCD study, so at least the journal's reputability is not relevant to quality of data collection and most of the experimental design. Edit: of course, this is not a rebuttal of your specific criticisms.
323 4y ago
Playing with toys does make kids smarter. It's not such a stretch to assume that video games have a benefit too.
- slowmovintarget 4y ago
  The right kind of toys, yes... and the right kind of video games. There are toys, and games, that make you dumber. Especially games designed to emphasize the addiction loop and monetize inconvenience. Case in point: Angry Birds. It used to be a very fun, silly, physics-based game. Now, it is infested with pay-to-unlock consumables that in some cases are required to get all three stars on a level (because you can't knock everything down without an explosion and the default roster of birds for the level doesn't give you that). The simpler a physical toy is (a ball, simple blocks) the more likely it is to contribute to a child's development. The insidious "I-need-another-outfit-Barbie" on the other hand only trains frivolous spending. Even Lego sets vary in the kind of play they foster. Playing with toys and games can have cognitive benefits, but, digital or otherwise, there's a quality spectrum parents have to be aware of.
  - hypertele-Xii 4y ago
    > Especially games designed to emphasize the addiction loop and monetize inconvenience. I think that falls within the purview of gambling.
jcranberry 4y ago
They also did a post hoc analysis using parental education instead of SES by the suggestion of a reviewer and the video game correlation persisted.
barbecue_sauce 4y ago
What about Sim City?
rapind 4y ago
> If you wanna believe that your hours on COD have made you a genius, go ahead, I won't stop you. Just don't imagine that this research proves it. Video games, like all things, should not all be treated equal. I could certainly see problem solving skills developing from world building or highly complex games (Civ, PoE, etc.). In fact, most (but not all) highly successful games have depth, which requires time investment and problem solving. The difference in games can be as varied as comparing a marketing pamphlet to Asimov's novels. I don't dispute your take on the quality of the research though. I would even go further and speculate it would be really really hard to come up with meaningful tests due to game variance. So most anything on the subject is likely fluff.
- the_only_law 4y ago
  > I could certainly see problem solving skills developing from world building or highly complex games (Civ, PoE, etc.). In fact, most (but not all) highly successful games have depth, which requires time investment and problem solving. I’m a big grand strategy fan, mostly Paradox games rn, and I almost feel like these are worse for me because the depth keeps me engaged longer (and honesty waste a lot of time) compared even to something like a shitty copy/paste mobile game employing dark patterns because those get so boring so quick. Whereas if I start and eu4 or ck3 campaign and actually play it, it’s almost certain my brain will be shot to hell to for a few days.
  - nluken 4y ago
    I love Paradox games, but I can't play them because as soon as I start up a campaign I know it will become a huge time sink for a few days. I think their games definitely require/develop problem solving skills, especially in areas where you need to make changes to create indirect effects. With that said, they may not be the best way of going about it.
  - boringg 4y ago
    Ahh I agree with this. I think there is a benefit at the beginning in the learning process of the game but then it has seriously diminishing returns. I would put factorio in this category where once you get the model and have learned most of the curve you are then at diminishing returns or negative returns because of the painful way in that devs construct these games to keep you on for longer periods of times. I'm not sure the highest levels of complexity and long dedication to the game really do provide greater learning for people.
jgalt212 4y ago
> In general, these guys have correlations, not causation You can say that for almost every human study that's not drug based, or very short-term.
- dash2 4y ago
  No. * There are tons of randomized controlled trials of policy measures (malaria bednets, minimum income). Many measure long-run outcomes. * Natural experiments can measure long-run effects. In economic history, sometimes that means centuries. * Many other designs are plausibly causal. The right instrumental variable, or a regression discontinuity design. In some cases, even a simple diff-in-diff with panel data. This design, nope.
  - pocketsand 4y ago
    The treatment here is regular social media use and video game playing. This not something where you can randomize people to treatment conditions, like you can with malaria. Likewise, tell me -- what would be a good instrument for estimating a causal effect of video game playing? What measure would plausibly affect intelligence only through video game playing? Where is there a natural experiment that allows for an RD design where young people on one side of the discontinuity play video games and the others do not? We get it -- you've taken a causal inference or econometrics course and want people to know it.
    - svnt 4y ago
      Waldorf schools or Amish communities would be a place to start.
    - dash2 4y ago
      No, my point is that bad research which conforms to people's priors should not be taken seriously. There are many experiments on video game playing. Most of them are short run, obviously. But never underestimate researcher ingenuity. Here's a cute paper which uses as an instrument "did your roommate bring a video game to university"? Not beyond critique, but plausible: https://economics.uwo.ca/people/stinebrickner_docs/paper2.pdf https://economics.uwo.ca/people/stinebrickner_docs/paper2.pd.... It's also relevant because the dependent variable is how much students study, and their resulting performance. More to the point, if anyone seriously thinks video games will raise kids' IQ, and can persuade funders of it, they could simply give the treatment group an Xbox. That would be expensive - say $20K-ish - but much cheaper than the benefit of an extra IQ point or 2, over millions of children.
lelanthran 4y ago
> If you wanna believe that your hours on COD have made you a genius, go ahead, I won't stop you. Just don't imagine that this research proves it. Maybe it's the games you play (CoD) that make imagine that game-playing develops no reasoning skills. Play something else (starcraft, for example).
- dash2 4y ago
  I never claimed anything about whether game-playing developed reasoning skills. Indeed, I explicitly said "believe that if you want to." I just said this research didn't prove it.
boloust 4y ago
Is there a reason to consider the use of a noisy signal as a strong sign of a low-quality study, without addressing whether the noise is correlated with the outcome? Do we think this study would be improved if it did not control for genetics at all?
- dash2 4y ago
  It wouldn't be any worse. The question isn't whether the noise is correlated with the outcome. The question is whether the denoised variable would be much more correlated with the outcome (yes), and whether that would change the effect of the focal variable. In any case, it's a weird thing to control for in a panel study. Why not just use per-person fixed effects? That would eliminate all effects that are constant across individuals.
DeathArrow 4y ago
Good points. If you'd want to find the impact of Digital Media on children's IQ, how would you proceed?
- dash2 4y ago
  A first shot would be a look at the introduction of digital media in particular places or times. Some people have done this with the internet by using e.g. distance to an ADSL exchange as an instrument. Or, a randomized controlled trial. If it has an effect worth caring about, then it's worth running an experiment on. A real positive or negative effect would be a big deal for policy.
- svnt 4y ago
  The question is too big and poorly defined, and IQ as an indicator is both incomplete and too narrow.
Foobar8568 4y ago
Pattern recognition+spacial rotation+quick answer to stimuli+repetitive tasks or short term memories, several IQ components are testing these. Kids might have higher IQ because of video games, but poorer to everything else that matters.
- riskable 4y ago
  > Pattern recognition+spacial rotation+quick answer to stimuli+repetitive tasks Sounds like we're training kids--through video games--to be fantastic CAD engineers and plumbers.
wing-_-nuts 4y ago
>If you wanna believe that your hours on COD have made you a genius, go ahead, I won't stop you. Just don't imagine that this research proves it. You know there are more video games styles than FPS right? Strategy games teach patience and discipline, EVE online teaches economics, even the much dismissed 'mindless' fps teach teamwork. I think it's likely that at least some games do increase intelligence relative to other activities (i.e. mindlessly watching tv) but less so than others like reading.
- dijit 4y ago
  > mindlessly I think this is probably more key than you think. Mindlessly watching junk food TV is not going to help you a lot. It's not very "nutrient rich" (to continue the expression) in terms of knowledge gained.. but you will probably gain some. Watching documentaries and, crucially, actively watching them is probably very good in terms of how much you learn.
  - wing-_-nuts 4y ago
    It takes a certain kind of nerdy kid to sit down and watch a documentary (not judging, I was that certain kind of nerdy kid). With video games, the user, i.e. child is actively engaged. Video games require more from the consumer. I found myself much more likely to dig into history while playing civilization than I did while watching a documentary for example.
    - svnt 4y ago
      Actively engaged does not mean exercising higher cognitive function, and even less so developing higher cognitive function. Imagine if e.g. basketball players never practiced but just played for all the marbles at the most competitive level every day when they stepped on the court. You would have chaos and exhausted players.
pocketsand 4y ago
If the noise in polygenic scores is random, all that will do is reduce correlations. Random measurement error always reduces the ability to observe relationships. To be clear, I do not know enough about polygenic scores to judge one way or another how the noise "works" vis-a-vis the entire analysis. The Porsche comment is snide, but actually exposes a similar error in your critique. Sure, a tax return-derived measure of income would be superior to measuring if someone owned a luxury car. But, if you found yourself in a situation where all you had to go on for measuring economic wellbeing was (luxury) car ownership, your analysis is likely to improve by including it rather than excluding it, unless the measure itself had serious other issues with its accuracy. Likewise, for SES, it is an imperfect measure, but it is the best we have for measuring social position in a concise way. Having worked in research and universities for a while, the type of critique presented in this post is one you often see of new graduate students. They are able to tear down problems with research very well, but tend to overlook whether the study itself was still informative, or whether the opposite finding is likely to be true. For example, suppose we wanted to know if video games or watching videos on the internet are making you dumber. A study like this may not convince you it's making you smarter, but it presents decent evidence they're not making you dumber. You can point out how the measures aren't perfect, but that is far from saying the opposite is true or the observed trends are completely spurious.
- dash2 4y ago
  The point is not whether including polygenic scores is better than nothing. The point is whether it's good enough to justify the claims they are making. It's not. The same holds for SES. I disagree that this study presents decent evidence of anything. I don't claim that the conclusions are false. But they haven't backed them up. There are lots of ways that the observed trends can be spurious. I mentioned some. The study is very weakly informative.
  - DANK_YACHT 4y ago
    > The point is not whether including polygenic scores is better than nothing. The point is whether it's good enough to justify the claims they are making. It's not. What is the justification for this assertion? If polygenic scores are simply "noisy," then, as the GP mentioned, they may be good enough when used in aggregate. There can be a lot of signal in noisy data. Ask any ML practitioner.
Mikeb85 4y ago
Anecdata but one thing I've noticed over time between those who grew up on video games and those who didn't is that those who were gamers deal better with change, especially WRT learning new software in the workplace, starting new projects, etc... Obviously though the benefits aren't there if people just mindlessly play the same game all day.
- riskable 4y ago
  This makes sense: Every game is a new piece of software that has to be installed/inserted with its own interface and mechanics. If you played a lot of games it means you had to get used to using a lot of different interfaces and controls.
asdff 4y ago
If you are going to critique the methodology please provide a reference where this is not a robust method. You may be right but how can I tell without some references? The scholar.google search for "controlling for genetics using polygenetic scores" brings up many recent papers about this methodology and the arguments made in these seem stronger to me than this one comment. IMO on the internet when people can easily misinterpret the science, its important to be clear as possible especially when we take things down. As far as scientific reports goes its a fine journal, its run by nature. It's not on the same planet as the predatory journals that spam inboxes. I worry that people will read your comment, assume you speak from authority, and discount any work they might see coming from that journal when we both know that good science can be found in scientific reports, and that impact factor is more strongly correlated with "sexy" or expensive science than good science anyhow.
- dash2 4y ago
  I don't speak from authority, but I do speak from experience as an academic. Scientific Reports is run by Springer, who are more competent than Elsevier but just as predatory. Their acceptance rate is 48%, and in 2021, they published 23000 articles. (You can check this here: https://www.nature.com/srep/research-articles?year=2021 https://www.nature.com/srep/research-articles?year=2021.) At a publication fee of almost $2000 (Wikipedia), this is a money-making machine, as skewered by https://www.youtube.com/watch?v=8F9gzQz1Pms https://www.youtube.com/watch?v=8F9gzQz1Pms. I think it's obvious what the incentives are here. The reason the method is not robust is that the typical polygenic score explains only 10% or less of the variance of its target phenotype. That leaves 90% of the variance unaccounted for by the control, which means your error term will be correlated with your focal dependent variable, violating the requirements for regression to give an unbiased estimate. I don't think these claims are controversial. We know polygenic scores are noisy. We know what happens when your control variables are noisy. The fact that lots of people do it doesn't, sadly, make it work. Lots of social psychologists run trials with an N of 35 (though they're addressing this critique, to their credit). Lots of historians fail to specify their hypotheses and to search for disconfirming data. Economists spent the 80s and 90s running cross-country regressions, before realizing that they had, in aggregate, more independent variables than cases. And so on.
_d2qh 4y ago
The worst things you can say about Scientific Reports are it doesn't have the same novelty/impact requirements as many journals* and people like to over-emphasize the "Nature" part of the name in a pointles grab for journal prestige. Journals in general can be pretty shady on a case-by-case basis. See "Nano Chopsticks" published in Nano Letters for a good materials science example of straight garbage (laughably obvious image manipulation) getting published in a fancy journal. *Not really a problem if you correctly believe science should be more than publishing sexy results.
epgui 4y ago
I don't really agree with your qualification of Scientific Reports as "not quite predatory" or "a very low bar". Though it doesn't come close to comparing to Nature or Science, it's a perfectly respectable journal and it publishes quality research.

by dash2 4y ago

nicce 4y ago

The cost is no joke. ”Scientific Reports is an open access journal. To publish in Scientific Reports, all authors are required to pay an article-processing charge (APC) of $1,495.”

napier 4y ago
“Open access” journal fees are often accounted for by institutional affiliations or grant funding donors. The upside of upfront publication charges is that the research output isn’t paywalled and therefore ideally reaches more eyeballs and brainpans than the conventional route.
- nicce 4y ago
  That is the upside, but there are many other high reputation peer-reviewed open journals with close to zero payment. Is the domain really worth the cost? It is kinda messed up, when you adjust the price based on the available money of institutional affiliations or donors. It is just more profits for journal and nothing for authors, while forcing authors to pay.
  - Aeolun 4y ago
    > there are many other high reputation peer-reviewed open journals with close to zero payment Which are those? The journals that I expect to still be up are the ones that require actual payment for publishing.
    - nicce 4y ago
      For example accepted papers in HICSS are stored publicly without extra cost. > Starting with HICSS-50, all HICSS publication will be archived and disseminated at no cost to all readers worldwide through ScholarSpace. A complete set of HICSS proceedings will also be made available in the digital library of the Association of Information Systems (AIS). https://hicss.hawaii.edu/resources/ https://hicss.hawaii.edu/resources/
- svnt 4y ago
  Most grants in the fields I’ve looked at preclude their use for paying publishing fees.
bigmadshoe 4y ago
This is becoming very common in academia. The university I used to work for is beginning to require that all research is published in open access journals. Do you know the ludicrous fees universities pay to closed-access journals?
BeetleB 4y ago
This is not an unusual charge for open access journals. The New Journal of Physics, a respected open access physics journal, charges $2225. https://publishingsupport.iopscience.iop.org/journals/new-journal-of-physics/about-new-journal-physics/#publication-charges https://publishingsupport.iopscience.iop.org/journals/new-jo...

throwingrocks 4y ago

Your response seemed reasonable, and I was nodding along until the last paragraph. I get the impression you had your mind made up about video games and intelligence long before this study was published.

lettergram 4y ago
I think the point the prior commenter was making was that this research validates people who think gaming is good. They pointed out the issues and said “if you still want to believe this, I can’t stop you” The commenter may have a bias, but most prior research shows us the opposite of the study. I also saw the poor experimental design and had a similar thoughts. Basically, this research looks poorly done and like an effort to prop up the gaming industry (and / or validate the authors pre-suppositions).
- d3ckard 4y ago
  In any case, using CoD as an example does not exactly show good will. I personally believe in huge gains from gaming, based on personal experience (so obviously n = 1, read further accordingly). Platformers train hand-eye coordination and pattern recognition, strategies teach resource management, RPGs about optimization and adopting growth mindset, racing games require long-time concentration, puzzlers and adventure games test your logic. In general, games require you to: - learn a set of tools - master them - conquer objective while also prevailing in face of adversity. I never regretted the hours I spent gaming and I feel they contributed very much to my softeng career (not directly though).
  - guerrilla 4y ago
    > good will. I think you meant good faith. They didn't use it as an example though, it was just rhetorical.
  - hackernewds 4y ago
    The question is about incrementality. in those same hours you played CoD, do you think (assuming the endpoint as you suggest is being a softeng) you could have learned more by practicing softeng (even if for half the time)

versteegen 4y ago

Note though that their data comes from the ABCD study, so at least the journal's reputability is not relevant to quality of data collection and most of the experimental design. Edit: of course, this is not a rebuttal of your specific criticisms.

323 4y ago

Playing with toys does make kids smarter. It's not such a stretch to assume that video games have a benefit too.

slowmovintarget 4y ago
The right kind of toys, yes... and the right kind of video games. There are toys, and games, that make you dumber. Especially games designed to emphasize the addiction loop and monetize inconvenience. Case in point: Angry Birds. It used to be a very fun, silly, physics-based game. Now, it is infested with pay-to-unlock consumables that in some cases are required to get all three stars on a level (because you can't knock everything down without an explosion and the default roster of birds for the level doesn't give you that). The simpler a physical toy is (a ball, simple blocks) the more likely it is to contribute to a child's development. The insidious "I-need-another-outfit-Barbie" on the other hand only trains frivolous spending. Even Lego sets vary in the kind of play they foster. Playing with toys and games can have cognitive benefits, but, digital or otherwise, there's a quality spectrum parents have to be aware of.
- hypertele-Xii 4y ago
  > Especially games designed to emphasize the addiction loop and monetize inconvenience. I think that falls within the purview of gambling.

jcranberry 4y ago

They also did a post hoc analysis using parental education instead of SES by the suggestion of a reviewer and the video game correlation persisted.

barbecue_sauce 4y ago

What about Sim City?

rapind 4y ago

> If you wanna believe that your hours on COD have made you a genius, go ahead, I won't stop you. Just don't imagine that this research proves it. Video games, like all things, should not all be treated equal. I could certainly see problem solving skills developing from world building or highly complex games (Civ, PoE, etc.). In fact, most (but not all) highly successful games have depth, which requires time investment and problem solving. The difference in games can be as varied as comparing a marketing pamphlet to Asimov's novels. I don't dispute your take on the quality of the research though. I would even go further and speculate it would be really really hard to come up with meaningful tests due to game variance. So most anything on the subject is likely fluff.

the_only_law 4y ago
> I could certainly see problem solving skills developing from world building or highly complex games (Civ, PoE, etc.). In fact, most (but not all) highly successful games have depth, which requires time investment and problem solving. I’m a big grand strategy fan, mostly Paradox games rn, and I almost feel like these are worse for me because the depth keeps me engaged longer (and honesty waste a lot of time) compared even to something like a shitty copy/paste mobile game employing dark patterns because those get so boring so quick. Whereas if I start and eu4 or ck3 campaign and actually play it, it’s almost certain my brain will be shot to hell to for a few days.
- nluken 4y ago
  I love Paradox games, but I can't play them because as soon as I start up a campaign I know it will become a huge time sink for a few days. I think their games definitely require/develop problem solving skills, especially in areas where you need to make changes to create indirect effects. With that said, they may not be the best way of going about it.
- boringg 4y ago
  Ahh I agree with this. I think there is a benefit at the beginning in the learning process of the game but then it has seriously diminishing returns. I would put factorio in this category where once you get the model and have learned most of the curve you are then at diminishing returns or negative returns because of the painful way in that devs construct these games to keep you on for longer periods of times. I'm not sure the highest levels of complexity and long dedication to the game really do provide greater learning for people.

jgalt212 4y ago

> In general, these guys have correlations, not causation You can say that for almost every human study that's not drug based, or very short-term.

dash2 4y ago
No. * There are tons of randomized controlled trials of policy measures (malaria bednets, minimum income). Many measure long-run outcomes. * Natural experiments can measure long-run effects. In economic history, sometimes that means centuries. * Many other designs are plausibly causal. The right instrumental variable, or a regression discontinuity design. In some cases, even a simple diff-in-diff with panel data. This design, nope.
- pocketsand 4y ago
  The treatment here is regular social media use and video game playing. This not something where you can randomize people to treatment conditions, like you can with malaria. Likewise, tell me -- what would be a good instrument for estimating a causal effect of video game playing? What measure would plausibly affect intelligence only through video game playing? Where is there a natural experiment that allows for an RD design where young people on one side of the discontinuity play video games and the others do not? We get it -- you've taken a causal inference or econometrics course and want people to know it.
  - svnt 4y ago
    Waldorf schools or Amish communities would be a place to start.
  - dash2 4y ago
    No, my point is that bad research which conforms to people's priors should not be taken seriously. There are many experiments on video game playing. Most of them are short run, obviously. But never underestimate researcher ingenuity. Here's a cute paper which uses as an instrument "did your roommate bring a video game to university"? Not beyond critique, but plausible: https://economics.uwo.ca/people/stinebrickner_docs/paper2.pdf https://economics.uwo.ca/people/stinebrickner_docs/paper2.pd.... It's also relevant because the dependent variable is how much students study, and their resulting performance. More to the point, if anyone seriously thinks video games will raise kids' IQ, and can persuade funders of it, they could simply give the treatment group an Xbox. That would be expensive - say $20K-ish - but much cheaper than the benefit of an extra IQ point or 2, over millions of children.

lelanthran 4y ago

> If you wanna believe that your hours on COD have made you a genius, go ahead, I won't stop you. Just don't imagine that this research proves it. Maybe it's the games you play (CoD) that make imagine that game-playing develops no reasoning skills. Play something else (starcraft, for example).

dash2 4y ago
I never claimed anything about whether game-playing developed reasoning skills. Indeed, I explicitly said "believe that if you want to." I just said this research didn't prove it.

boloust 4y ago

Is there a reason to consider the use of a noisy signal as a strong sign of a low-quality study, without addressing whether the noise is correlated with the outcome? Do we think this study would be improved if it did not control for genetics at all?

dash2 4y ago
It wouldn't be any worse. The question isn't whether the noise is correlated with the outcome. The question is whether the denoised variable would be much more correlated with the outcome (yes), and whether that would change the effect of the focal variable. In any case, it's a weird thing to control for in a panel study. Why not just use per-person fixed effects? That would eliminate all effects that are constant across individuals.

DeathArrow 4y ago

Good points. If you'd want to find the impact of Digital Media on children's IQ, how would you proceed?

dash2 4y ago
A first shot would be a look at the introduction of digital media in particular places or times. Some people have done this with the internet by using e.g. distance to an ADSL exchange as an instrument. Or, a randomized controlled trial. If it has an effect worth caring about, then it's worth running an experiment on. A real positive or negative effect would be a big deal for policy.
svnt 4y ago
The question is too big and poorly defined, and IQ as an indicator is both incomplete and too narrow.

Foobar8568 4y ago

Pattern recognition+spacial rotation+quick answer to stimuli+repetitive tasks or short term memories, several IQ components are testing these. Kids might have higher IQ because of video games, but poorer to everything else that matters.

riskable 4y ago
> Pattern recognition+spacial rotation+quick answer to stimuli+repetitive tasks Sounds like we're training kids--through video games--to be fantastic CAD engineers and plumbers.

wing-_-nuts 4y ago

>If you wanna believe that your hours on COD have made you a genius, go ahead, I won't stop you. Just don't imagine that this research proves it. You know there are more video games styles than FPS right? Strategy games teach patience and discipline, EVE online teaches economics, even the much dismissed 'mindless' fps teach teamwork. I think it's likely that at least some games do increase intelligence relative to other activities (i.e. mindlessly watching tv) but less so than others like reading.

dijit 4y ago
> mindlessly I think this is probably more key than you think. Mindlessly watching junk food TV is not going to help you a lot. It's not very "nutrient rich" (to continue the expression) in terms of knowledge gained.. but you will probably gain some. Watching documentaries and, crucially, actively watching them is probably very good in terms of how much you learn.
- wing-_-nuts 4y ago
  It takes a certain kind of nerdy kid to sit down and watch a documentary (not judging, I was that certain kind of nerdy kid). With video games, the user, i.e. child is actively engaged. Video games require more from the consumer. I found myself much more likely to dig into history while playing civilization than I did while watching a documentary for example.
  - svnt 4y ago
    Actively engaged does not mean exercising higher cognitive function, and even less so developing higher cognitive function. Imagine if e.g. basketball players never practiced but just played for all the marbles at the most competitive level every day when they stepped on the court. You would have chaos and exhausted players.

pocketsand 4y ago

If the noise in polygenic scores is random, all that will do is reduce correlations. Random measurement error always reduces the ability to observe relationships. To be clear, I do not know enough about polygenic scores to judge one way or another how the noise "works" vis-a-vis the entire analysis. The Porsche comment is snide, but actually exposes a similar error in your critique. Sure, a tax return-derived measure of income would be superior to measuring if someone owned a luxury car. But, if you found yourself in a situation where all you had to go on for measuring economic wellbeing was (luxury) car ownership, your analysis is likely to improve by including it rather than excluding it, unless the measure itself had serious other issues with its accuracy. Likewise, for SES, it is an imperfect measure, but it is the best we have for measuring social position in a concise way. Having worked in research and universities for a while, the type of critique presented in this post is one you often see of new graduate students. They are able to tear down problems with research very well, but tend to overlook whether the study itself was still informative, or whether the opposite finding is likely to be true. For example, suppose we wanted to know if video games or watching videos on the internet are making you dumber. A study like this may not convince you it's making you smarter, but it presents decent evidence they're not making you dumber. You can point out how the measures aren't perfect, but that is far from saying the opposite is true or the observed trends are completely spurious.

dash2 4y ago
The point is not whether including polygenic scores is better than nothing. The point is whether it's good enough to justify the claims they are making. It's not. The same holds for SES. I disagree that this study presents decent evidence of anything. I don't claim that the conclusions are false. But they haven't backed them up. There are lots of ways that the observed trends can be spurious. I mentioned some. The study is very weakly informative.
- DANK_YACHT 4y ago
  > The point is not whether including polygenic scores is better than nothing. The point is whether it's good enough to justify the claims they are making. It's not. What is the justification for this assertion? If polygenic scores are simply "noisy," then, as the GP mentioned, they may be good enough when used in aggregate. There can be a lot of signal in noisy data. Ask any ML practitioner.

Mikeb85 4y ago

Anecdata but one thing I've noticed over time between those who grew up on video games and those who didn't is that those who were gamers deal better with change, especially WRT learning new software in the workplace, starting new projects, etc... Obviously though the benefits aren't there if people just mindlessly play the same game all day.

riskable 4y ago
This makes sense: Every game is a new piece of software that has to be installed/inserted with its own interface and mechanics. If you played a lot of games it means you had to get used to using a lot of different interfaces and controls.

asdff 4y ago

If you are going to critique the methodology please provide a reference where this is not a robust method. You may be right but how can I tell without some references? The scholar.google search for "controlling for genetics using polygenetic scores" brings up many recent papers about this methodology and the arguments made in these seem stronger to me than this one comment. IMO on the internet when people can easily misinterpret the science, its important to be clear as possible especially when we take things down. As far as scientific reports goes its a fine journal, its run by nature. It's not on the same planet as the predatory journals that spam inboxes. I worry that people will read your comment, assume you speak from authority, and discount any work they might see coming from that journal when we both know that good science can be found in scientific reports, and that impact factor is more strongly correlated with "sexy" or expensive science than good science anyhow.

dash2 4y ago
I don't speak from authority, but I do speak from experience as an academic. Scientific Reports is run by Springer, who are more competent than Elsevier but just as predatory. Their acceptance rate is 48%, and in 2021, they published 23000 articles. (You can check this here: https://www.nature.com/srep/research-articles?year=2021 https://www.nature.com/srep/research-articles?year=2021.) At a publication fee of almost $2000 (Wikipedia), this is a money-making machine, as skewered by https://www.youtube.com/watch?v=8F9gzQz1Pms https://www.youtube.com/watch?v=8F9gzQz1Pms. I think it's obvious what the incentives are here. The reason the method is not robust is that the typical polygenic score explains only 10% or less of the variance of its target phenotype. That leaves 90% of the variance unaccounted for by the control, which means your error term will be correlated with your focal dependent variable, violating the requirements for regression to give an unbiased estimate. I don't think these claims are controversial. We know polygenic scores are noisy. We know what happens when your control variables are noisy. The fact that lots of people do it doesn't, sadly, make it work. Lots of social psychologists run trials with an N of 35 (though they're addressing this critique, to their credit). Lots of historians fail to specify their hypotheses and to search for disconfirming data. Economists spent the 80s and 90s running cross-country regressions, before realizing that they had, in aggregate, more independent variables than cases. And so on.

_d2qh 4y ago

The worst things you can say about Scientific Reports are it doesn't have the same novelty/impact requirements as many journals* and people like to over-emphasize the "Nature" part of the name in a pointles grab for journal prestige. Journals in general can be pretty shady on a case-by-case basis. See "Nano Chopsticks" published in Nano Letters for a good materials science example of straight garbage (laughably obvious image manipulation) getting published in a fancy journal. *Not really a problem if you correctly believe science should be more than publishing sexy results.

epgui 4y ago

I don't really agree with your qualification of Scientific Reports as "not quite predatory" or "a very low bar". Though it doesn't come close to comparing to Nature or Science, it's a perfectly respectable journal and it publishes quality research.