4 ms·

>our current tests don’t even come to measuring it. Then why do they correlate with outcomes so well ?

runarberg 5y ago
Maybe because they are designed to and because of selection bias. These tests are expected to correlate, if they don’t they are considered a bad test and is thrown into obscurity (but more likely fixed until it does).
- moonchrome 5y ago
  But if they didn't even come close to measuring it then there would be no correlation ? It's not like these tests are applied retroactively to explain past performances - they offer actual predictive value. Selection bias can account for some of it but there are ways to test if this and from what I've seen it still holds up.
  - runarberg 5y ago
    I’m not disputing that these tests measure something, they obviously do. However what they are measuring is not what most people consider a satisfactory definition of intelligence. In fact most psychometricians simply give up and define intelligence based on what these tests measure (i.e. operation definition). Without justification that is very poor science.
  - plorkyeran 5y ago
    SAT scores would be positively correlated with success even if they were entirely random as long as universities continued to use them as a factor in admissions. It's sort of a corollary of Goodhart's law that it is difficult to assess the quality of a measure while it is being used as a target.
jcranmer 5y ago
IIRC, the strongest predictive factor for SAT scores is... parental income. So if what these tests are measuring is not a "general factor of intelligence" but "quality of home environment", you can readily see how that could correlate well with outcomes like "success in later life."
- moonchrome 5y ago
  SAT mixes in intelligence with education so it's bound to skew more towards people with access, but it's also correlated with IQ, and IQ has significant heritability as well (things like separated twin studies have shown the genetic component quite clearly). So it's probably a double whammy there - good environment and good genetics.

Maybe because they are designed to and because of selection bias. These tests are expected to correlate, if they don’t they are considered a bad test and is thrown into obscurity (but more likely fixed until it does).

moonchrome 5y ago
But if they didn't even come close to measuring it then there would be no correlation ? It's not like these tests are applied retroactively to explain past performances - they offer actual predictive value. Selection bias can account for some of it but there are ways to test if this and from what I've seen it still holds up.
- runarberg 5y ago
  I’m not disputing that these tests measure something, they obviously do. However what they are measuring is not what most people consider a satisfactory definition of intelligence. In fact most psychometricians simply give up and define intelligence based on what these tests measure (i.e. operation definition). Without justification that is very poor science.
- plorkyeran 5y ago
  SAT scores would be positively correlated with success even if they were entirely random as long as universities continued to use them as a factor in admissions. It's sort of a corollary of Goodhart's law that it is difficult to assess the quality of a measure while it is being used as a target.

jcranmer 5y ago

IIRC, the strongest predictive factor for SAT scores is... parental income. So if what these tests are measuring is not a "general factor of intelligence" but "quality of home environment", you can readily see how that could correlate well with outcomes like "success in later life."

moonchrome 5y ago
SAT mixes in intelligence with education so it's bound to skew more towards people with access, but it's also correlated with IQ, and IQ has significant heritability as well (things like separated twin studies have shown the genetic component quite clearly). So it's probably a double whammy there - good environment and good genetics.