5 ms·

> “A high test score tells you only that a person has high ability and is a good match for that particular test at that point in time,” says Matthews. If Matth

by trendia 10y ago

> “A high test score tells you only that a person has high ability and is a good match for that particular test at that point in time,” says Matthews. If Matthews is correct that scores are transient, then we'd expect test scores to have little to no predictive ability. In other words, we'd expect the rate of career achievement among high test takers to be approximately equal to the rate of achievement among low test takers. But this claim is directly contradicted by actual evidence: > Follow-up surveys—at ages 18, 23, 33 and 48—backed up his hunch. A 2013 analysis found a correlation between the number of patents and peer-refereed publications that people had produced and their earlier scores on SATs and spatial-ability tests. The SAT tests jointly accounted for about 11% of the variance; spatial ability accounted for an additional 7.6%.

jsprogrammer 10y ago
It's a huge stretch to consider patents and peer-refereed publications "actual evidence". Others have taken other approaches and found no correlations.
- trendia 10y ago
  There were five different features that they plotted vs. SAT score [1]. All five rely heavily on intelligence: 1. any doctorate 2. STEM publications 3. STEM doctorates 4. Patents 5. Income in the 95th percentile A person in Q4 was 60% more likely than a person in Q3 to earn income in the 95th percentile. I'd say that's pretty significant. [1] https://www.scientificamerican.com/sciam/assets/Image/Child_genius_chart_newsfeature_web.jpg https://www.scientificamerican.com/sciam/assets/Image/Child_...
  - jsprogrammer 10y ago
    I believe that is only evidence of bias towards high score holders, which we already know exists. Minor issue: why does that figures' caption lead with a quip about the top 1%, but only present data for quartiles?
    - hyperpape 10y ago
      I'm not sure what those Q1-Q4 numbers are, but I don't think they are quartiles. I went to a TIP ceremony for kids who'd scored well on the SATs in 7th grade. There were a few hundred kids there (I think something like 1000 got you there?), a few dozen with 1200 something, and one kid up on stage with a 1400.
      - jsprogrammer 10y ago
        The bottom right of the figure says, "[s]tudents were split into quartiles". I don't get why there is a reference to the top 1% in the caption though.
        MagnumOpus 10y ago
        Step 1: Take highly able kids who score in the top 1% in entrance exams or can get SAT scores of 700 at age 13. Step 2: Sort these highly able kids into quartiles by their test scores. Result: Even within the top 1% of IQ, the relative ability still matters. I.e. there is no "cutoff" for becoming a great scientist or inventor; smarter is always better.
    - trendia 10y ago
      IQ measured while young predicts income later as an entrepreneur. http://imgur.com/a/huYKk http://imgur.com/a/huYKk http://ftp.iza.org/dp3648.pdf http://ftp.iza.org/dp3648.pdf
  - dalke 10y ago
    Features #1-3 do not "rely heavily on intelligence". While intelligence certainly has a role, they rely much more heavily on training. All three also depend on knowing what is interesting, which is more a form of curiosity guided by knowing what other people know than a form of intelligence that can be measured by taking the SAT. (I don't know enough about #4 or #5 to make a comment.)
    - trendia 10y ago
      People with graduate degrees have higher than average IQ, but people with higher IQ do not necessarily have graduate degrees. That is, intelligence is a necessary (but not sufficient) condition to get a graduate degree in STEM. The other necessary conditions you mention (training, interest, etc.) are also important, but still require a baseline ability. Even looking at undergraduate degrees, IQ is a significant factor in what degree is obtained. Physics and astronomy majors have an average of IQ [1] of over 130 -- that's 2 standard deviations above the mean. If interest, training, and hard work alone were sufficient to get a physics degree, we'd see a much lower average IQ among physics students. [1] http://www.randalolson.com/2014/06/25/average-iq-of-students-by-college-major-and-gender-ratio/ http://www.randalolson.com/2014/06/25/average-iq-of-students...
      - jsprogrammer 10y ago
        Again, you are only exposing group bias. Doctorates are largely predicated on test scores - they are a filter on who can access a defense committee - of course those with doctorates will have higher prior test scores; those with high test score were intentionally picked as doctoral candidates, in part, because of their test score(s).
        trendia 10y ago
        What I think you're trying to say is that intelligence is irrelevant to getting a Doctorate because test scores, not intelligence, are the factor. But of course doctorates are based on test scores. That's because test scores are an indicator of intelligence, and so admissions committees filter based on test scores in order to filter on intelligence. So that means yes, admissions depends on test scores, but no, test scores are not independent of intelligence. That is: P(high IQ | high test score) > P(low IQ | high test score) What you're trying to say is that getting a doctorate a degree doesn't depend on intelligence, and is just an indicator of some "group bias". What I'm saying is that test scores are correlated with IQ. This implies that if you do not have high intelligence, it is going to be extremely difficult to get the test scores required to get into graduate school, and thus intelligence is required for admission. In other words, intelligence is indeed a necessary but not sufficient condition to get a graduate degree in STEM.
- dredmorbius 10y ago
  What alternatives would you propose? Why?
jmde 10y ago
Depending on how you define it, you could say my research is in this general area. My sense of how to interpret these sorts of findings has changed greatly over time, and become much more moderate and humble. Discussions over these issues tend to be very heated and people oversimplify issues on both ends. These tests are predictive, and as you point out, it's disingenuous to claim otherwise. They're measuring something valid. However, they're doing so relatively poorly, or rather, roughly. Even the quote you cite suggests that the total amount of variance accounted for was about 19%, which isn't too shabby by any means, but is also very crude when you think about implications for real-world consequences. Interpreting that 19% also becomes very murky, when you consider factors like teaching to tests, that tests like those are used for academic and career selection, leading to self-fulfilling prophecies. In rare circumstances where researchers have been able to remove some of these effects (for example, due to unusual legal decisions about acceptable use of test scores), the relationship between test scores and career performance has been shown to be very weak in the short-term, and almost non-existent in the long term. Somewhere underlying the distribution of test scores is some distribution of ability that makes it easier for some people to thrive in certain settings. But ability is just one thing in a long list of things, along with practice, resources and other environmental factors that contribute to success, and sociological dynamics. Add to that test scores being very crude indices of ability to begin with, and you're in a very messy situation. What's frustrating to me is that there's this sort of all-or-nothing attitude about testing, where tests are seen as useless, or as a perfect indicator of skill or ability. The truth is in-between.
- AStellersSeaCow 10y ago
  >What's frustrating to me is that there's this sort of all-or-nothing attitude about testing, where tests are seen as useless, or as a perfect indicator of skill or ability. The truth is in-between. Yes. This is a major point people need to focus on. I think, to bastardize a famous quote, that testing is the worst simple measure of aptitude, except all the others we've tried. In a perfect world, any admissions/hiring decision would be made after multiple independent in-depth examinations of a candidate's past work, plus extensive interviews. Since that level of vetting is unfeasible for all but a tiny sliver of the most desirable spots, we need ballpark estimates like testing that will be more predictive than coin flips.
sokoloff 10y ago
I am part of the SMPY study cohort (from 1982) and have another close friend who is from the 1983 or 1984 study cohort. (The qualification at the time was 700 or greater on the SAT Math before age 13. This was, of course "SAT one" scoring system.) I also spent some in summer camps with other students in the study and have remained casual friends with some of them over the years. I don't see any evidence to suggest that that level of demonstrated ability is transient.
blahblah3 10y ago
“A high test score tells you only that a person has high ability and is a good match for that particular test at that point in time,” says Matthews. I have never understood the intention behind this type of statement (which I've seen many variants of commonly espoused). Let X be the event "got a high test score at a particular test at a certain point in time." Is Matthews saying X = X (an uninteresting tautology)? Or is he claiming that X is uncorrelated or not predictive of other variables of interest (career success, ability on other task) which is clearly empirically false.
- dredmorbius 10y ago
  My read is that expertise carries with it specificity. If you happen to be very good at some X[3], but a test is only effectively measuring ability in X[5], where 3 & 5 are some arbitrary specialisations of a capability X, then your test score may not be particularly high. This depends on just how specific a skill is, whether or not the particular subject is a polymath or specialist, and much more. To take an example from athletics, Lance Armstrong was a world-leading cyclist (with, it turns out, the advantage of doping). When he entered into marathon events -- for which his light body frame is generally still an advantage -- he placed respectably, but not as one of the very top finishers. A skills or abilities test for Armstrong had to be specific to his ~60 km cycling distance specialisation. There is some crossover of many skills, but the deep specialisation, ingrained structures, algorithmic knowledge, solution sapces, bags-of-tricks, etc., that are associated with being best in class (not merely "very good") can be quite complex.
  - blahblah3 10y ago
    That's a great example and I don't disagree with you. The way I think about "intelligence" is not to mull over what it "really means" but to think about lots of common "intellectual" tasks (in which aptitude can be measured) that people are interested in: reading, writing, math, programming, solving puzzles, etc. Or ideally, theoretical "potential" in each of these tasks, since one person may put more or less effort into another field. What's supported by research and anectodal evidence is that if you did principal components analysis on these measured "aptitudes", there would be a principal component explaining a lot of the variation. A good test can try to measure this principal component. But when you're talking about the very highest levels of specific tasks like cycling or swimming, it would be reasonable to expect that amongst the population of already "elite athletes" there would be less of a dominant first principal component

by trendia 10y ago

jsprogrammer 10y ago

It's a huge stretch to consider patents and peer-refereed publications "actual evidence". Others have taken other approaches and found no correlations.

trendia 10y ago
There were five different features that they plotted vs. SAT score [1]. All five rely heavily on intelligence: 1. any doctorate 2. STEM publications 3. STEM doctorates 4. Patents 5. Income in the 95th percentile A person in Q4 was 60% more likely than a person in Q3 to earn income in the 95th percentile. I'd say that's pretty significant. [1] https://www.scientificamerican.com/sciam/assets/Image/Child_genius_chart_newsfeature_web.jpg https://www.scientificamerican.com/sciam/assets/Image/Child_...
- jsprogrammer 10y ago
  I believe that is only evidence of bias towards high score holders, which we already know exists. Minor issue: why does that figures' caption lead with a quip about the top 1%, but only present data for quartiles?
  - hyperpape 10y ago
    I'm not sure what those Q1-Q4 numbers are, but I don't think they are quartiles. I went to a TIP ceremony for kids who'd scored well on the SATs in 7th grade. There were a few hundred kids there (I think something like 1000 got you there?), a few dozen with 1200 something, and one kid up on stage with a 1400.
    - jsprogrammer 10y ago
      The bottom right of the figure says, "[s]tudents were split into quartiles". I don't get why there is a reference to the top 1% in the caption though.
      - MagnumOpus 10y ago
        Step 1: Take highly able kids who score in the top 1% in entrance exams or can get SAT scores of 700 at age 13. Step 2: Sort these highly able kids into quartiles by their test scores. Result: Even within the top 1% of IQ, the relative ability still matters. I.e. there is no "cutoff" for becoming a great scientist or inventor; smarter is always better.
  - trendia 10y ago
    IQ measured while young predicts income later as an entrepreneur. http://imgur.com/a/huYKk http://imgur.com/a/huYKk http://ftp.iza.org/dp3648.pdf http://ftp.iza.org/dp3648.pdf
- dalke 10y ago
  Features #1-3 do not "rely heavily on intelligence". While intelligence certainly has a role, they rely much more heavily on training. All three also depend on knowing what is interesting, which is more a form of curiosity guided by knowing what other people know than a form of intelligence that can be measured by taking the SAT. (I don't know enough about #4 or #5 to make a comment.)
  - trendia 10y ago
    People with graduate degrees have higher than average IQ, but people with higher IQ do not necessarily have graduate degrees. That is, intelligence is a necessary (but not sufficient) condition to get a graduate degree in STEM. The other necessary conditions you mention (training, interest, etc.) are also important, but still require a baseline ability. Even looking at undergraduate degrees, IQ is a significant factor in what degree is obtained. Physics and astronomy majors have an average of IQ [1] of over 130 -- that's 2 standard deviations above the mean. If interest, training, and hard work alone were sufficient to get a physics degree, we'd see a much lower average IQ among physics students. [1] http://www.randalolson.com/2014/06/25/average-iq-of-students-by-college-major-and-gender-ratio/ http://www.randalolson.com/2014/06/25/average-iq-of-students...
    - jsprogrammer 10y ago
      Again, you are only exposing group bias. Doctorates are largely predicated on test scores - they are a filter on who can access a defense committee - of course those with doctorates will have higher prior test scores; those with high test score were intentionally picked as doctoral candidates, in part, because of their test score(s).
      - trendia 10y ago
        What I think you're trying to say is that intelligence is irrelevant to getting a Doctorate because test scores, not intelligence, are the factor. But of course doctorates are based on test scores. That's because test scores are an indicator of intelligence, and so admissions committees filter based on test scores in order to filter on intelligence. So that means yes, admissions depends on test scores, but no, test scores are not independent of intelligence. That is: P(high IQ | high test score) > P(low IQ | high test score) What you're trying to say is that getting a doctorate a degree doesn't depend on intelligence, and is just an indicator of some "group bias". What I'm saying is that test scores are correlated with IQ. This implies that if you do not have high intelligence, it is going to be extremely difficult to get the test scores required to get into graduate school, and thus intelligence is required for admission. In other words, intelligence is indeed a necessary but not sufficient condition to get a graduate degree in STEM.
dredmorbius 10y ago
What alternatives would you propose? Why?

jmde 10y ago

Depending on how you define it, you could say my research is in this general area. My sense of how to interpret these sorts of findings has changed greatly over time, and become much more moderate and humble. Discussions over these issues tend to be very heated and people oversimplify issues on both ends. These tests are predictive, and as you point out, it's disingenuous to claim otherwise. They're measuring something valid. However, they're doing so relatively poorly, or rather, roughly. Even the quote you cite suggests that the total amount of variance accounted for was about 19%, which isn't too shabby by any means, but is also very crude when you think about implications for real-world consequences. Interpreting that 19% also becomes very murky, when you consider factors like teaching to tests, that tests like those are used for academic and career selection, leading to self-fulfilling prophecies. In rare circumstances where researchers have been able to remove some of these effects (for example, due to unusual legal decisions about acceptable use of test scores), the relationship between test scores and career performance has been shown to be very weak in the short-term, and almost non-existent in the long term. Somewhere underlying the distribution of test scores is some distribution of ability that makes it easier for some people to thrive in certain settings. But ability is just one thing in a long list of things, along with practice, resources and other environmental factors that contribute to success, and sociological dynamics. Add to that test scores being very crude indices of ability to begin with, and you're in a very messy situation. What's frustrating to me is that there's this sort of all-or-nothing attitude about testing, where tests are seen as useless, or as a perfect indicator of skill or ability. The truth is in-between.

AStellersSeaCow 10y ago
>What's frustrating to me is that there's this sort of all-or-nothing attitude about testing, where tests are seen as useless, or as a perfect indicator of skill or ability. The truth is in-between. Yes. This is a major point people need to focus on. I think, to bastardize a famous quote, that testing is the worst simple measure of aptitude, except all the others we've tried. In a perfect world, any admissions/hiring decision would be made after multiple independent in-depth examinations of a candidate's past work, plus extensive interviews. Since that level of vetting is unfeasible for all but a tiny sliver of the most desirable spots, we need ballpark estimates like testing that will be more predictive than coin flips.

sokoloff 10y ago

I am part of the SMPY study cohort (from 1982) and have another close friend who is from the 1983 or 1984 study cohort. (The qualification at the time was 700 or greater on the SAT Math before age 13. This was, of course "SAT one" scoring system.) I also spent some in summer camps with other students in the study and have remained casual friends with some of them over the years. I don't see any evidence to suggest that that level of demonstrated ability is transient.

blahblah3 10y ago

“A high test score tells you only that a person has high ability and is a good match for that particular test at that point in time,” says Matthews. I have never understood the intention behind this type of statement (which I've seen many variants of commonly espoused). Let X be the event "got a high test score at a particular test at a certain point in time." Is Matthews saying X = X (an uninteresting tautology)? Or is he claiming that X is uncorrelated or not predictive of other variables of interest (career success, ability on other task) which is clearly empirically false.

dredmorbius 10y ago
My read is that expertise carries with it specificity. If you happen to be very good at some X[3], but a test is only effectively measuring ability in X[5], where 3 & 5 are some arbitrary specialisations of a capability X, then your test score may not be particularly high. This depends on just how specific a skill is, whether or not the particular subject is a polymath or specialist, and much more. To take an example from athletics, Lance Armstrong was a world-leading cyclist (with, it turns out, the advantage of doping). When he entered into marathon events -- for which his light body frame is generally still an advantage -- he placed respectably, but not as one of the very top finishers. A skills or abilities test for Armstrong had to be specific to his ~60 km cycling distance specialisation. There is some crossover of many skills, but the deep specialisation, ingrained structures, algorithmic knowledge, solution sapces, bags-of-tricks, etc., that are associated with being best in class (not merely "very good") can be quite complex.
- blahblah3 10y ago
  That's a great example and I don't disagree with you. The way I think about "intelligence" is not to mull over what it "really means" but to think about lots of common "intellectual" tasks (in which aptitude can be measured) that people are interested in: reading, writing, math, programming, solving puzzles, etc. Or ideally, theoretical "potential" in each of these tasks, since one person may put more or less effort into another field. What's supported by research and anectodal evidence is that if you did principal components analysis on these measured "aptitudes", there would be a principal component explaining a lot of the variation. A good test can try to measure this principal component. But when you're talking about the very highest levels of specific tasks like cycling or swimming, it would be reasonable to expect that amongst the population of already "elite athletes" there would be less of a dominant first principal component