6 ms·
Ok so I see no controls for income/family wealth, etc If anything I see selection for wealth: the samples in one of the bigger studies selected from “the 15 be
by olliej 7y ago
Ok so I see no controls for income/family wealth, etc
If anything I see selection for wealth: the samples in one of the bigger studies selected from “the 15 best STEM graduate training programmed”, which is broadly a proxy for wealth.
More over they don’t have negative samples: afaict their “study” was to take the top X% of students from some already wealth biased institutes and compare their performance to the overall average for everyone. What they should have done is compare the performance of the X% of the class they sampled from with the performance of the remainder of that class.
Then we could compare more similar starting points.
Obviously we would generally expect some correlation between skill when younger to 35 years later, but I would expect the entire group to do significantly better than the general average. Eg a person who got into Harvard because of their family history, and gets a C, is likely to do better than a person with a 4.0 from a state university. (General averages, obviously there can be outliers in every direction)
- iandanforth 7y agoThis is junk science that can best be used to reinforce existing stratification or create/justify a self-fulfilling prophecy of success. Best to ignore it and move on.
- deleted 7y ago[deleted]
- burpsnard 7y agoTheir method would miss the likes of Blake, Van Gogh, Galileo, ...
- sokoloff 7y agoSMPY identifies outlier ability via standardized testing taken before age 13 (for cohorts other than the very last one). All of Study 1 was selected this way. This may have some very loose correlation with wealth. Is there a correlation between SMPY eligibility and getting into elite universities? Probably, but it doesn’t seem like an improper one. Is there a correlation between elite universities and life outcome? Sure. Does that make the longitudinal study from age 11/12 into adulthood useless or content-free? Not in my opinion. Only Study 2 was done on cohort 5, which was selected as grad students already at elite universities.
- olliej 7y agoCohort 4 was the only state university sample, but even that one used students in the summer programme (which is tied to wealth). I also realized that SAT scores are heavily impacted by SAT specific training/tutoring that costs a reasonable amount (because as far as I can tell - as a foreigner - the SAT tests rely more on having been taught the exact way - and tricks - to answer SAT questions in the time available. Honestly if your test is good it should not have any real time pressure for the vast majority of participants)
- thaumasiotes 7y ago> I also realized that SAT scores are heavily impacted by SAT specific training/tutoring It's difficult to realize something that isn't true. The effects of training on SAT scores have always been negligible, less than the measurement error associated with taking the test one time.
- sokoloff 7y agoMost people scoring 700+ on SAT-M experience no time pressure on the test. (That could be read as tautological, but I’m fairly convinced that there is no real time pressure that would dramatically shift the “SMPY/SET eligible” from not. The questions are straightforward and there’s more than ample time provided for those with a realistic chance of scoring 700+.) The extent of my tutoring was a public school guidance counselor explaining to our whole class who was taking the test that there was statistically no penalty for guessing and how the “A is greater, B is greater, C means equal, D means indeterminate” questions worked. That was 30 minutes at most.
- confidantlake 7y agoI got a 760 on SAT-M and I definitely felt tremendous time pressure. I am convinced I could have gotten 800 with no time limit. At the same time someone much smarter than me could probably get an 800 with half the time given.
- Spooky23 7y agoMy high school did intense drilling for SATs as an after school thing. The average kid in my class had a 180 point improvement versus the PSAT as a result. High SAT scores are evidence of high SAT score.
- boomboomsubban 7y ago>Obviously we would generally expect some correlation between skill when younger to 35 years later Though this is obvious, it wasn't obvious that the tests given were actually showing who had some amounts of skill. Though, this should more accurately be called "some amount of ability to become prominent in a field," as like you say things other than skill may cause both high scores and success later.
- gwern 7y agoThe justification for using the SAT-M in underage testing is in fact that: because the material would not have yet been covered in any classes, it tests ability to come up with a correct answer to a novel problem rather than skill. (Remember, educators have long been convinced that any kind of acceleration of gifted kids is poison, and SMPY's results have been a major reason that gifted kids can do any acceleration these days, so they're not just covering it in some accelerated math class.)
- boomboomsubban 7y agoThey could be covering the material with private tutors or similar outside the classroom experience. That's why we can't say for certain the test is measuring skill.
- knzhou 7y agoUh oh, that seems to be rather out of date. When I took the SAT-M in 7th grade I recall having already seen harder versions of all the questions, because of math competitions. The same went for almost everyone I knew that took it.
- abecedarius 7y agoFWIW I took it in the 80s (also around 7th grade though I forget exactly) and it was not like that for me. I wish I'd had more contact with real math that early -- school was such a waste of time.
- gwern 7y ago
- ahupp 7y agoThe way I read the study is that within the top N% group, higher scores predict "eminence". Not the top N compared to everyone else.
- knzhou 7y ago> If anything I see selection for wealth: the samples in one of the bigger studies selected from “the 15 best STEM graduate training programmed”, which is broadly a proxy for wealth. Not everything is about wealth. STEM graduate school doesn't select for wealth at all. Undergraduate admissions sometimes do, but essentially zero of the wealth-based admits major in STEM. In fact in my experience the effect is the exact opposite of what you say: STEM takes in whoever can do the work, regardless of wealth, and helps produce new wealth. In every institution I've been in, STEM people had the lowest average family wealth by far.
- erosenbe0 7y agoIt would be interesting to compare the trajectories of the bottom 5% of Harvard grads with the top 5% of UMass grads. As a thought exercise, you can ask yourself: would you prefer that your doctor be from the former or the latter group?
- bkohlmann 7y agoDefinitely the latter. Bottom 5% of Harvard are likely legacies with little motivation - and unlikely to get into medical school.
- gwern 7y ago> Ok so I see no controls for income/family wealth, etc There is no reason to control for those, and doing so would be an 'Everest regression': "controlling for altitude, Mount Everest is actually warm today". What you are asking for is like reading about IQ and income and saying "ah, but did they control for education?" Controlling is not magic pixie dust.
- davidgerard 7y ago> What you are asking for is like reading about IQ and income and saying "ah, but did they control for education?" that thing that literally raises the measured IQ number? you know that practice can raise your score, right
- davidmanheim 7y agoRight - but there is also a relationship between IQ and educational attainment. So controlling for it means that you're measuring something fundamentally different. That's fine, but you can't just control for things without having a valid theoretical construct to justify it. (Well, OK, you can, and people do, and it's great for getting p<0.05 if you don't care about getting meaningful answers about the underlying questions.)
- georgehemingto1 7y agoit's a triviality to say that you can control for educational attainment (and opportunities for education as well as class) when accounting for differences in IQ: it's hard to do in practice because social science is complicated and requires ACTUALLY DOING THE SCIENCE so gwern's glib comment is just wrong here clearly we're talking about the impact on IQ of whatever factors we make up to describe IQ, and obviously, and provenly, education (and therefore the ability to get education) is in play here, which gwern's glib rebuttal ignores
- georgehemingto1 7y agohonestly it's disappointing when i see people with training in stats make these comments: I'm a fucking poet if I'm anything at all and yet I find myself drawn into these ridiculous discussions about stats (in which I have no formal training) where I feel like I'm one of the only people in the room who knows how to divy up numbers