5 ms·
This topic is a very good example of the risks inherent in trying to do science by consensus. The study [1] this article references had a huge sample size (19,9
by rjf72 8y ago
This topic is a very good example of the risks inherent in trying to do science by consensus. The study [1] this article references had a huge sample size (19,957) and is also probably very replicable. It's also complete trash. The methodology was to call and survey parents of 2-5 year olds asking questions such as on a (1/never to 5/always) 'How would you rate your child's interest and curiosity in learning new things' or 'How would your child's laughing and smiling tendencies'?
They then proceed to massage the data in a bunch of different ways, in no small part because they acknowledge that their variables ended up providing no decent mapping of reliability - in the study's words, "these variables were not combined into a composite well‐being measure because the reliability of these items was relatively low." And after this massaging and subjective "normalizing" they conclude that TV and video games are just awesome for 2-5 year olds.
Ultimately on complex topics that cannot be objectively and meaningfully tested you can generally prove whatever you want to prove. To actually have something vaguely resembling meaning here you'd need to take a random sampling of children and then require that one group provide no access to digitech, another provide 1-2 hours of digitech, and the final group 5+ hours or whatever. Do this for years and then compare the children using objective measurements determined by a blind third party or examination. The random factor there is also required. If you just survey households that chose to allow their children e.g. 0 or 5+ hours of digitech per day, then you're going to get some severe environmental and perhaps genetic confounding variables in your results. In any case no study like this is ever going to happen in modern times.
And granted even that sort of study would have a million possible flaws, but studies such as the one this article are just jokes. And they are very much the rule rather than the exception. We tend to attribute quality to something with a doctorate or two behind it, but publish or perish alongside let's say 'motivated' research is increasingly distorting science.
[1] - https://onlinelibrary.wiley.com/doi/full/10.1111/cdev.13007 https://onlinelibrary.wiley.com/doi/full/10.1111/cdev.13007
- flukus 8y ago> This topic is a very good example of the risks inherent in trying to do science by consensus. Just to be clear, I'm not talking about doing science but communicating it. Even just cataloging the various studies and assigning some sort of label for reliability like you just did would be valuable. As you acknowledge, there is an awful lot of noise, but as a layman I have no way of separating the signal from the noise.
- rjf72 8y agoOf course, but what you're requesting is science. It's a meta-analysis. A work that studies other studies and tries to provide aggregate data while controlling for whatever the meta-analysis creators determine to be relevant. What I was pointing out here is how studies like this would generally be considered high-reliability studies for these meta-analyses. It ticks all the right boxes, probably by design. Yet at the same time when you look at it, it's something I think few would find worthy of basing their decisions on. But qualifying exactly which studies you should, or should not, give weight to is a task that's ultimately going to also involve substantial bias. If I want to argue that digitech is good for kids, I'm going to be able to create very reasonable sounding criteria that would include studies such as this. If I want to argue that digitech is bad for kids, I'm still going to be able to create very reasonable sounding criteria yet one that would exclude studies such as this. The same problem inherent in the individual studies also ends up emerging in the meta-analyses as well. Aggregating data in some ways takes you even further away from the truth. Here you can reasonably show that this study is not so great, yet imagine this was part of hundreds of different studies all packed together with a sample size aggregate of hundreds of thousands. It may very well be that that meta-analysis was also loaded with dubious work, such as this, yet it becomes vastly more tedious to illustrate such. And as the rate of publication continues to exponentially increase, then going forward into the future we will eventually be faced with things such as meta-analyses of meta-analyses. Meta-meta-analysis I suppose?