5 ms·

The actual paper, available on sci-hub, does not agree with this innumerate journalist's interpretation of it. http://www.pnas.org.sci-hub.ac/content/113/47/13

The actual paper, available on sci-hub, does not agree with this innumerate journalist's interpretation of it. http://www.pnas.org.sci-hub.ac/content/113/47/13354.short?rss=1 http://www.pnas.org.sci-hub.ac/content/113/47/13354.short?rs... In fact, the paper shows that IQ is a great predictor. Personality is a much worse predictor on highly g-loaded tasks, and a good predictor on some others (e.g. wages). But (IQ, personality) pairs are an even better predictor than IQ or personality. The only way you can draw the conclusion that IQ doesn't matter much is if you compare to predictors of the form (IQ, personality, achievement, grades) (and a few similar ones). Shockingly, using achievement to predict achievement yields a more accurate predictor than using (IQ, personality). (I really do not understand this paper's methodology of graphing r^2 values. I don't get how this was published, but lets assume that was a reasonable way to go.) This paper also ignores the fact that IQ, grades, and many personality traits are correlated. Insofar as these metrics are correlated, it completely invalidates this analysis; some of what the authors attribute to personality or grades is actually just the explanatory power of a second IQ measurement.

tomrod 10y ago
I haven't read the paper yet, but I'd be genuinely surprised if PCA or factor analysis wasn't performed.
- yummyfajitas 10y ago
  I just skimmed it, but I don't see them doing that anywhere.
Declanomous 10y ago
They are testing whether personality and IQ are correlated to success. R-squared is a measurement of dependence. So basically they are graphing this correlation on a scale from from 0 being "this variable is independent of success", to 1 being "success is perfectly correlated with this variable". The actual measurements are in a separate appendix, so I'm not sure how the calculated all this.
mcguire 10y ago
Check Fig. 4 and the discussion: "Column 1 in Fig. 4 in the first block of columns (corresponding to wages) shows that IQ predicts wages, but the predictive power is small (around 1% [Adj. R squared of 0.02 in Fig. 4). Column 2 in Fig. 4 shows that self-esteem, locus of control, antisocial behavior, and neuroticism, taken together, are more important determinants of wages [R sq. Of .05]. Both IQ and personality remain as important predictors in wage equations when both are included in a regression (column 3 in Fig. 4). The fourth column in Fig. 4 shows that achievement has more predictive power than IQ and personality alone. When IQ and personality are also included in a regression (column 5 in Fig. 4), achievement test scores remain an important predictor of wages, and IQ and personality also remain important predictors of wages. After controlling for scores on achievement tests, IQ loses around 60% of its predictive power. When grades are included, instead of achievement tests, the effect of IQ becomes negligible. A similar pattern arises across the other outcomes studied." The achievement measure is from a test (of learned facts and skills?) at age 10 (IIUC). Further, the paper seems to go to some effort to decompose the raw data into IQ, grades. See table 2. The correlation between iq and personality is fairly weak.
- yummyfajitas 10y ago
  The problem is that achievement tests and grades are a measure of IQ. Imagine I have a direct relationship, y = g. Now I measure compute a noisy data set, (x,y) with x = g + noise. Depending on the size of the noise I'll discover that y is correlated with g with some R^2. My regression coefficient will come out to be approximately y = 1x. Now imagine I made a new noisy data set, (x1, x2, x3, y). Here x1 = g + noise, x2 = g + noise, x3 = g + noise. Here x1 might be IQ, x2 might be SAT or other such g-loaded test, x3 grades, etc. My new regression coefficients are y = (1/3)x1 + (1/3)x2 + (1/3)x3. Suddenly x1 has lost 2/3 of it's predictive power! The problem is that my measurements are correlated, that's all. And as you note, table 2 shows a pretty solid correlation between IQ, personality, achievement and grades. If you actually want to draw the conclusion that the article is trying to draw, you need to decouple IQ from the other measurements.
  - Godel_unicode 10y ago
    In order for them to be directly related as you suggest, all high-iq participants need to memorize lots of facts and learn lots of skills and no low-iq participants can do similarly. It seems that, by citing them as separate factors, the authors are saying this is not true for the population they considered.
GFK_of_xmaspast 10y ago
I think, as a Bayesian, you're putting too much weight on your priors about the importance of IQ and are not interpreting the paper correctly. "Achievement" in the context of the paper refers to scores on achievement tests, whereas the thing of actual interest is 'life outcome', and the analysis of that starts on page 3.
kome 10y ago
This doesn't deserve to be the top comment.
hsitz 10y ago
After reading your comment, I was surprised to take a look at the paper and find that it does, indeed, appear to support conclusion that IQ is less important than many people think. Neither the paper nor the linked article argue that IQ is not a factor. But both say that other factors are more important, this is clear from both the abstract and the first paragraph of the article, where it says, "Personality is generally more predictive than IQ on a variety of important life outcomes." Of course, personality+IQ will be more predictive than either alone. But the "innumerate" journalist seems to me to have done a better job of capturing the gist of the paper than you. (Whether the paper's methodology and analysis are sound is a different question from whether the journalist reported its findings correctly.)
- yummyfajitas 10y ago
  I don't know what "less important than many people think" means. It's true that the article suggests people think 50% of variance is explained by IQ, and it turns out that literally nothing in this paper explains even 25% of variance. On log-wages, IQ explains 30-50% of the explained variance depending on whether you go by figure 4 or 5. It does a similar job for education and welfare usage. So it seems that if people have any misconceptions, it's merely about how much of the variance in log wages at age 40 is explainable by early life measurements.
gwern 10y ago
I would add that the procedure of using a plain multivariate regression to draw quasi-causal conclusions is definitely wrong because of the fallacy of controlling for intermediate variables (IQ causes education, so an estimate of the form log(wages) ~ IQ + education is not estimating what you think it is estimating and a small estimate for IQ is expected), and they seem to be very casual about the measurement error though they know it's driving a lot of the results: "For example, in the NLSY79, IQ is a better predictor of log wages than personality, but in the BCS and the MIDUS data, personality measures are better predictors. The better and more comprehensive personality measures in the BCS and the MIDUS data compared with those available in the NLSY data likely explain why personality is more predictive of outcomes in those data." Not to mention that correlations with income tend to increase when you look at multi-year averages and unbin & untruncate the single-year income data to boot. If they did a more meaningful SEM (like Education ~ IQ, Wages ~ IQ + Education), with correction for all the measurement error, I would expect that, in line with the rest of the literature (I have collected some links on the topic in http://www.gwern.net/Embryo%20selection#iqincome-bibliography http://www.gwern.net/Embryo%20selection#iqincome-bibliograph... ), they would find that IQ outpredicts the rest of the variables, followed by whatever corresponds to Conscientiousness.

tomrod 10y ago

I haven't read the paper yet, but I'd be genuinely surprised if PCA or factor analysis wasn't performed.

yummyfajitas 10y ago
I just skimmed it, but I don't see them doing that anywhere.

Declanomous 10y ago

They are testing whether personality and IQ are correlated to success. R-squared is a measurement of dependence. So basically they are graphing this correlation on a scale from from 0 being "this variable is independent of success", to 1 being "success is perfectly correlated with this variable". The actual measurements are in a separate appendix, so I'm not sure how the calculated all this.

mcguire 10y ago

Check Fig. 4 and the discussion: "Column 1 in Fig. 4 in the first block of columns (corresponding to wages) shows that IQ predicts wages, but the predictive power is small (around 1% [Adj. R squared of 0.02 in Fig. 4). Column 2 in Fig. 4 shows that self-esteem, locus of control, antisocial behavior, and neuroticism, taken together, are more important determinants of wages [R sq. Of .05]. Both IQ and personality remain as important predictors in wage equations when both are included in a regression (column 3 in Fig. 4). The fourth column in Fig. 4 shows that achievement has more predictive power than IQ and personality alone. When IQ and personality are also included in a regression (column 5 in Fig. 4), achievement test scores remain an important predictor of wages, and IQ and personality also remain important predictors of wages. After controlling for scores on achievement tests, IQ loses around 60% of its predictive power. When grades are included, instead of achievement tests, the effect of IQ becomes negligible. A similar pattern arises across the other outcomes studied." The achievement measure is from a test (of learned facts and skills?) at age 10 (IIUC). Further, the paper seems to go to some effort to decompose the raw data into IQ, grades. See table 2. The correlation between iq and personality is fairly weak.

yummyfajitas 10y ago
The problem is that achievement tests and grades are a measure of IQ. Imagine I have a direct relationship, y = g. Now I measure compute a noisy data set, (x,y) with x = g + noise. Depending on the size of the noise I'll discover that y is correlated with g with some R^2. My regression coefficient will come out to be approximately y = 1x. Now imagine I made a new noisy data set, (x1, x2, x3, y). Here x1 = g + noise, x2 = g + noise, x3 = g + noise. Here x1 might be IQ, x2 might be SAT or other such g-loaded test, x3 grades, etc. My new regression coefficients are y = (1/3)x1 + (1/3)x2 + (1/3)x3. Suddenly x1 has lost 2/3 of it's predictive power! The problem is that my measurements are correlated, that's all. And as you note, table 2 shows a pretty solid correlation between IQ, personality, achievement and grades. If you actually want to draw the conclusion that the article is trying to draw, you need to decouple IQ from the other measurements.
- Godel_unicode 10y ago
  In order for them to be directly related as you suggest, all high-iq participants need to memorize lots of facts and learn lots of skills and no low-iq participants can do similarly. It seems that, by citing them as separate factors, the authors are saying this is not true for the population they considered.

GFK_of_xmaspast 10y ago

I think, as a Bayesian, you're putting too much weight on your priors about the importance of IQ and are not interpreting the paper correctly. "Achievement" in the context of the paper refers to scores on achievement tests, whereas the thing of actual interest is 'life outcome', and the analysis of that starts on page 3.

kome 10y ago

This doesn't deserve to be the top comment.

hsitz 10y ago

After reading your comment, I was surprised to take a look at the paper and find that it does, indeed, appear to support conclusion that IQ is less important than many people think. Neither the paper nor the linked article argue that IQ is not a factor. But both say that other factors are more important, this is clear from both the abstract and the first paragraph of the article, where it says, "Personality is generally more predictive than IQ on a variety of important life outcomes." Of course, personality+IQ will be more predictive than either alone. But the "innumerate" journalist seems to me to have done a better job of capturing the gist of the paper than you. (Whether the paper's methodology and analysis are sound is a different question from whether the journalist reported its findings correctly.)

yummyfajitas 10y ago
I don't know what "less important than many people think" means. It's true that the article suggests people think 50% of variance is explained by IQ, and it turns out that literally nothing in this paper explains even 25% of variance. On log-wages, IQ explains 30-50% of the explained variance depending on whether you go by figure 4 or 5. It does a similar job for education and welfare usage. So it seems that if people have any misconceptions, it's merely about how much of the variance in log wages at age 40 is explainable by early life measurements.

gwern 10y ago

I would add that the procedure of using a plain multivariate regression to draw quasi-causal conclusions is definitely wrong because of the fallacy of controlling for intermediate variables (IQ causes education, so an estimate of the form log(wages) ~ IQ + education is not estimating what you think it is estimating and a small estimate for IQ is expected), and they seem to be very casual about the measurement error though they know it's driving a lot of the results: "For example, in the NLSY79, IQ is a better predictor of log wages than personality, but in the BCS and the MIDUS data, personality measures are better predictors. The better and more comprehensive personality measures in the BCS and the MIDUS data compared with those available in the NLSY data likely explain why personality is more predictive of outcomes in those data." Not to mention that correlations with income tend to increase when you look at multi-year averages and unbin & untruncate the single-year income data to boot. If they did a more meaningful SEM (like Education ~ IQ, Wages ~ IQ + Education), with correction for all the measurement error, I would expect that, in line with the rest of the literature (I have collected some links on the topic in http://www.gwern.net/Embryo%20selection#iqincome-bibliography http://www.gwern.net/Embryo%20selection#iqincome-bibliograph... ), they would find that IQ outpredicts the rest of the variables, followed by whatever corresponds to Conscientiousness.