6 ms·

From the article: > Note that IQ tests correlate with SAT scores! That is (in part) Nassim Nicholas Taleb's point: we use IQ to select for success, then say I

From the article: > Note that IQ tests correlate with SAT scores! That is (in part) Nassim Nicholas Taleb's point: we use IQ to select for success, then say IQ is correlated with success. It is circular reasoning. The more interesting measure is not jobs for which IQ is the main selection criteria to qualify for the training (your examples of MDs, JDs, and PhDs), but things where people create their own impact, lets say entrepreneurs, or world changing ideas (think Naler's Nudge). I don't necessarily agree with this, BTW. Just linking the ideas back together.

naniwaduni 8y ago
> That is (in part) Nassim Nicholas Taleb's point: we use IQ to select for success, then say IQ is correlated with success. It is circular reasoning. This would only be correct if the SAT were expressly an IQ test, and we selected people based on SAT score for that reason. To put it another way, making this point is kind of like selecting for tall players for a basketball team, then saying weight is correlated with fitness to be on a basketball team. That height and weight are correlated doesn't make this circular reasoning.
- harry8 8y ago
  Yeah totally, or like selecting for confident people where you tell people who score well on the test that they're super confident. Tall is an absolute, objective measure of a real thing that exists. IQ is NOT a measure of a thing that exists. Similar effects can be found throwing dice and ascribing the total to an IQ score, then telling parents, teachers what the poor kid's IQ is as though it were their height. It's insane.
  - harry8 8y ago
    http://homepages.gac.edu/~jwotton2/PSY225/rosenthal66.pdf http://homepages.gac.edu/~jwotton2/PSY225/rosenthal66.pdf https://sites.google.com/site/7arosenthal/ https://sites.google.com/site/7arosenthal/ Per: Rosenthal & Jacobson, 1966 study, which deserves wider appreciation? I want to emphasise 1966 was when this was published so it isn't overlooked as we all read pretty fast with these things. edit: Found a link to the original paper: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.488.8077&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.488... More recent criticism of the original: https://thenewstatistics.com/itns/2018/04/03/weve-been-here-before-the-replication-crisis-over-the-pygmalion-effect/ https://thenewstatistics.com/itns/2018/04/03/weve-been-here-...
  - feanaro 8y ago
    If it is not a real thing, how does it succeed in having an influence over results on a measurement device? Results on cognitive tests have a relatively high test-retest reliability.
  - wolfgangK 8y ago
    > IQ is NOT a measure of a thing that exists. I don't understand your concept of something "existing" or not. Do you believe that a "speed" exist ? That a "center of gravity" exists ? That a "temperature" exist ? People who say "IQ is what IQ tests measure" sound to me like they could say "temperature is what thermometers measure". temperature does not "exist" it is molecules movements all the way down, yet it is a meaningful/useful concept. Same goes for IQ or "g factor".
- chillacy 8y ago
  This seems like nit picking, doesn’t really change the patent posters argument.
waterhouse 8y ago
Hmm... perhaps. But I'm pretty sure that the questions of e.g. the law exam are about laws, and the questions of medical school exams are about anatomy and medicinal procedures—not about the "which shape completes the pattern" puzzles from IQ tests. I'm quite certain they are not given IQ tests. The SAT, meanwhile, asks math and English questions. Is Taleb saying that IQ is a bad metric, or is he saying that these other tests are academic and unrealistic, and then referring to them collectively as IQ tests because the results are correlated? Or something else?
- barry-cotter 8y ago
  IQ tests attempt to measure g, which is some combination of processing speed, capability for abstract thought and working memory, all of those being highly correlated. The GRE and GMAT are just harder versions of the SAT, IQ tests for people who completed middle school. The LSAT is an IQ test with no math portion > It is designed to assess reading comprehension as well as logical and verbal reasoning proficiency. https://en.wikipedia.org/wiki/Law_School_Admission_Test https://en.wikipedia.org/wiki/Law_School_Admission_Test The MCAT has one pure IQ test section out of four. However, all of these tests are highly g loaded. People who can do well on one of them can do well on the others with more or less preparation. That’s what g attempts to measure.
- roel_v 8y ago
  "But I'm pretty sure that the questions of e.g. the law exam are about laws" I don't know about mcat, but no, lsat isn't about that at all. It would be weird testing people for the thing they're trying to be admitted to. If you google 'lsat sample questions', it"s trivial to find what sort of thing is beimg tested for. Which is not 'complete the pattern', but closer to that (I would argue) than to substantive law questions.
  - waterhouse 8y ago
    Fascinating. I didn't realize that was how law school admission worked. By "law exam" I actually had in mind "the final exam" and whatever other tests you have to pass in order to not be thrown out of the program, once you're already in. (It seems the "bar exam" is not a prerequisite for a JD, though.) I guess the question is, which is more selective: law school admissions, or "who actually gets their degree once they're in law school"? Looks like the latter filter lets about one third through [1], while the former seems to pass one half if you just divide total admissions by total applications [2], although if we care more about "top law schools" I see various Google results saying it's more like 25%. So... reasonably similar-sized filters. Huh. I also see an article titled "Law Schools With Low LSAT Medians Have Absurd Academic Attrition Rates" [3]. That would seem to support the notion that the LSAT genuinely is testing ability to do well at law, despite being a non-law exam. [1] https://www.americanbar.org/content/dam/aba/administrative/legal_education_and_admissions_to_the_bar/statistics/enrollment_degrees_awarded.pdf https://www.americanbar.org/content/dam/aba/administrative/l... [2] https://www.americanbar.org/groups/legal_education/resources/statistics/ https://www.americanbar.org/groups/legal_education/resources... , see "JD Applicant/Matriculant Data" for e.g. 2016, which shows 351k applications and 145k acceptances (I had to type in "=SUM(B7:B210)" and "=SUM(C7:C210)" to get those totals) [3] https://abovethelaw.com/2018/01/law-schools-with-low-lsat-medians-had-absurd-academic-attrition-rates/?rf=1 https://abovethelaw.com/2018/01/law-schools-with-low-lsat-me...
asabjorn 8y ago
If the SAT test wasn't selective in some meaningful way then people admitted through affirmative action should be expected to do approximately as well as the people that was admitted purely through merit. The outcome says otherwise [1], or if prefer a video [2]. Mismatching people to schools based upon an ideal hurts those people when they come unprepared relative to their peers. Many diverse people admitted through affirmative action with a dream to do the sciences abandon them for other less rigorous disciplines, and it doesn't have to be so if they were matched with a school without the social engineering attempts of ideologues. [1] https://www.theatlantic.com/national/archive/2012/10/the-painful-truth-about-affirmative-action/263122/ https://www.theatlantic.com/national/archive/2012/10/the-pai... [2] https://www.youtube.com/watch?v=VVvnTByzTmA https://www.youtube.com/watch?v=VVvnTByzTmA
- LordDragonfang 8y ago
  Focussing on "affirmative action" feels uncomfortably close to the exact kind of dog whistling this article criticized up front. As I understand it, most colleges have been using some form of linear regression model on many factors to filter admissions in a way that maximize returns (i.e. alumni donations) for decades now. Surely there is other evidence which shows that if SAT scores were actually a poor heuristic of potential success, schools would have stopped relying on them by now. (without having to bring race into the conversation)
  - samatman 8y ago
    He didn't bring race into the conversation; you did. Affirmative action is precisely an adjustment of admissions criteria such that ceterus paribus a candidate is more likely to be admitted. Yes, it's based on race, but if it was based on lottery, it would have the same effect: students with lower SAT scores are admitted to schools which otherwise require a certain threshold. That they don't do as well, shows that the SAT is measuring something relevant to success in education. Note that affirmative action would have this effect even if average IQ were the same across races, because schools vary in the SAT thresholds they set, and this would have the effect of propelling students into settings they aren't prepared to flourish in.
pgeorgi 8y ago
> > Note that IQ tests correlate with SAT scores! IQ tests (of which there are a ton) are regularly calibrated, internationally comparable (there are some reasonably culturally neutral test protocols) and stable while SAT is a US-only metric. Maybe SAT is calibrated to correlate with IQ tests?
- barry-cotter 8y ago
  The SAT is not calibrated to correlate with IQ tests. Being correlated with IQ tests is an unavoidable consequence of what it is. It’s a high stakes test with limited subject matter meant to measure scholastic aptitude. That will unavoidably be highly g loaded. It’s not culture fair, like Raven’s Matrices, because it assumes you speak English at a native level and have been exposed to a middle school curriculum.
  - asabjorn 8y ago
    It sounds fair to assume some basic skills for admittance, such as a completed middle school competency level and native level proficiency in the language advanced university level material will be taught in. Any person without those will be severely disadvantaged relative to the peers that don’t.
hhjinks 8y ago
> we use IQ to select for success We do? I have never seen IQ used for anything but online wankery. None of my most skilled coworkers have struck me as people outside the bell curve, and we have never had our IQ test results requested, or even measured in most cases.
- barry-cotter 8y ago
  The SAT, SSAT, GRE, MCAT, LSAT and GMAT are all glorified IQ tests. They’re used for entrance to selective colleges and graduate programmes. Lots of people will be quite happy to tell you where they went to school and what they studied which is the non-vulgar way of signalling your IQ. Intelligence:Knowns and Unknowns > are used for many purposes, such as selection, diagnosis, and evaluation. Many of the most widely used tests are not intended to measure intelligence itself but some closely related construct: scholastic aptitude, school achievement, specific abilities, etc. Such tests are es- pecially important for selection purposes. For preparatory school, it's the SSAT; for college, the SAT or ACT; for graduate school, the GRE; for medical school, the MCAT; for law school, the LSAT; for business school, the GMAT. Scores on intelligence-related tests matter, and the stakes can be high. http://differentialclub.wdfiles.com/local--files/definitions-structure-and-measurement/Intelligence-Knowns-and-unknowns.pdf http://differentialclub.wdfiles.com/local--files/definitions...
  - gaius 8y ago
    I scored very well on the GMAT and it’s really testing how diligently you prepare for it, and that is the predictor of success, not the content of the test itself, IMHO.
    - barry-cotter 8y ago
      I’m sure you did improve. Practice effects exist. They’re also limited. Just as I will get faster if I train to run the 100m or run a marathon you can get better at the GMAT, LSAT etc. But there’s a plateau you hit where you can’t get better. If you try to take someone from 50th percentile to 99th on the GMAT in the majority of cases you will fail. An increase of 0.26 standard deviations isn’t nothing but on the GMAT it’s only ~30 points. https://digitalcommons.ilr.cornell.edu/cgi/viewcontent.cgi?article=1012&amp=&context=articles&amp=&sei-redir=1&referer=https%253A%252F%252Fscholar.google.com%252Fscholar%253Fhl%253Den%2526as_sdt%253D0%25252C5%2526q%253Dgmat%252Bpractice%252Beffect%2526btnG%253D#search=%22gmat%20practice%20effect%22 https://digitalcommons.ilr.cornell.edu/cgi/viewcontent.cgi?a... Retesting in selection: a meta-analysis of coaching and practice effects for tests of cognitive ability. Previous studies have indicated that as many as 25% to 50% of applicants in organizational and educational settings are retested with measures of cognitive ability. Researchers have shown that practice effects are found across measurement occasions such that scores improve when these applicants retest. In this study, the authors used meta-analysis to summarize the results of 50 studies of practice effects for tests of cognitive ability. Results from 107 samples and 134,436 participants revealed an adjusted overall effect size of 0.26. Moderator analyses indicated that effects were larger when practice was accompanied by test coaching and when identical forms were used. Additional research is needed to understand the impact of retesting on the validity inferences drawn from test scores.
commandlinefan 8y ago
> we use IQ to select for success, then say IQ is correlated with success. We don’t, though. The most financially successful people in the world are sports stars, movie stars, and business owners, none of whose success has anything to do with IQ scores. He bangs on IQ, but nobody actually uses IQ to select for anything – it’s actually prohibited by law in the US (https://www.britannica.com/event/Griggs-v-Duke-Power-Co https://www.britannica.com/event/Griggs-v-Duke-Power-Co). We do use SAT scores to determine admission to college, and SAT scores correlate pretty well with IQ scores, though, so I guess he’s indirectly complaining about SAT scores being used for college admissions? There’s nothing particularly new about this, though – people have complained for eons that measuring people’s test scores only tracks how good they are at taking tests. The problem is, true or not, tests are the only objective way we’ve ever come up with to figure out if somebody actually knows something.

naniwaduni 8y ago

> That is (in part) Nassim Nicholas Taleb's point: we use IQ to select for success, then say IQ is correlated with success. It is circular reasoning. This would only be correct if the SAT were expressly an IQ test, and we selected people based on SAT score for that reason. To put it another way, making this point is kind of like selecting for tall players for a basketball team, then saying weight is correlated with fitness to be on a basketball team. That height and weight are correlated doesn't make this circular reasoning.

harry8 8y ago
Yeah totally, or like selecting for confident people where you tell people who score well on the test that they're super confident. Tall is an absolute, objective measure of a real thing that exists. IQ is NOT a measure of a thing that exists. Similar effects can be found throwing dice and ascribing the total to an IQ score, then telling parents, teachers what the poor kid's IQ is as though it were their height. It's insane.
- harry8 8y ago
  http://homepages.gac.edu/~jwotton2/PSY225/rosenthal66.pdf http://homepages.gac.edu/~jwotton2/PSY225/rosenthal66.pdf https://sites.google.com/site/7arosenthal/ https://sites.google.com/site/7arosenthal/ Per: Rosenthal & Jacobson, 1966 study, which deserves wider appreciation? I want to emphasise 1966 was when this was published so it isn't overlooked as we all read pretty fast with these things. edit: Found a link to the original paper: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.488.8077&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.488... More recent criticism of the original: https://thenewstatistics.com/itns/2018/04/03/weve-been-here-before-the-replication-crisis-over-the-pygmalion-effect/ https://thenewstatistics.com/itns/2018/04/03/weve-been-here-...
- feanaro 8y ago
  If it is not a real thing, how does it succeed in having an influence over results on a measurement device? Results on cognitive tests have a relatively high test-retest reliability.
- wolfgangK 8y ago
  > IQ is NOT a measure of a thing that exists. I don't understand your concept of something "existing" or not. Do you believe that a "speed" exist ? That a "center of gravity" exists ? That a "temperature" exist ? People who say "IQ is what IQ tests measure" sound to me like they could say "temperature is what thermometers measure". temperature does not "exist" it is molecules movements all the way down, yet it is a meaningful/useful concept. Same goes for IQ or "g factor".
chillacy 8y ago
This seems like nit picking, doesn’t really change the patent posters argument.

waterhouse 8y ago

Hmm... perhaps. But I'm pretty sure that the questions of e.g. the law exam are about laws, and the questions of medical school exams are about anatomy and medicinal procedures—not about the "which shape completes the pattern" puzzles from IQ tests. I'm quite certain they are not given IQ tests. The SAT, meanwhile, asks math and English questions. Is Taleb saying that IQ is a bad metric, or is he saying that these other tests are academic and unrealistic, and then referring to them collectively as IQ tests because the results are correlated? Or something else?

barry-cotter 8y ago
IQ tests attempt to measure g, which is some combination of processing speed, capability for abstract thought and working memory, all of those being highly correlated. The GRE and GMAT are just harder versions of the SAT, IQ tests for people who completed middle school. The LSAT is an IQ test with no math portion > It is designed to assess reading comprehension as well as logical and verbal reasoning proficiency. https://en.wikipedia.org/wiki/Law_School_Admission_Test https://en.wikipedia.org/wiki/Law_School_Admission_Test The MCAT has one pure IQ test section out of four. However, all of these tests are highly g loaded. People who can do well on one of them can do well on the others with more or less preparation. That’s what g attempts to measure.
roel_v 8y ago
"But I'm pretty sure that the questions of e.g. the law exam are about laws" I don't know about mcat, but no, lsat isn't about that at all. It would be weird testing people for the thing they're trying to be admitted to. If you google 'lsat sample questions', it"s trivial to find what sort of thing is beimg tested for. Which is not 'complete the pattern', but closer to that (I would argue) than to substantive law questions.
- waterhouse 8y ago
  Fascinating. I didn't realize that was how law school admission worked. By "law exam" I actually had in mind "the final exam" and whatever other tests you have to pass in order to not be thrown out of the program, once you're already in. (It seems the "bar exam" is not a prerequisite for a JD, though.) I guess the question is, which is more selective: law school admissions, or "who actually gets their degree once they're in law school"? Looks like the latter filter lets about one third through [1], while the former seems to pass one half if you just divide total admissions by total applications [2], although if we care more about "top law schools" I see various Google results saying it's more like 25%. So... reasonably similar-sized filters. Huh. I also see an article titled "Law Schools With Low LSAT Medians Have Absurd Academic Attrition Rates" [3]. That would seem to support the notion that the LSAT genuinely is testing ability to do well at law, despite being a non-law exam. [1] https://www.americanbar.org/content/dam/aba/administrative/legal_education_and_admissions_to_the_bar/statistics/enrollment_degrees_awarded.pdf https://www.americanbar.org/content/dam/aba/administrative/l... [2] https://www.americanbar.org/groups/legal_education/resources/statistics/ https://www.americanbar.org/groups/legal_education/resources... , see "JD Applicant/Matriculant Data" for e.g. 2016, which shows 351k applications and 145k acceptances (I had to type in "=SUM(B7:B210)" and "=SUM(C7:C210)" to get those totals) [3] https://abovethelaw.com/2018/01/law-schools-with-low-lsat-medians-had-absurd-academic-attrition-rates/?rf=1 https://abovethelaw.com/2018/01/law-schools-with-low-lsat-me...

asabjorn 8y ago

If the SAT test wasn't selective in some meaningful way then people admitted through affirmative action should be expected to do approximately as well as the people that was admitted purely through merit. The outcome says otherwise [1], or if prefer a video [2]. Mismatching people to schools based upon an ideal hurts those people when they come unprepared relative to their peers. Many diverse people admitted through affirmative action with a dream to do the sciences abandon them for other less rigorous disciplines, and it doesn't have to be so if they were matched with a school without the social engineering attempts of ideologues. [1] https://www.theatlantic.com/national/archive/2012/10/the-painful-truth-about-affirmative-action/263122/ https://www.theatlantic.com/national/archive/2012/10/the-pai... [2] https://www.youtube.com/watch?v=VVvnTByzTmA https://www.youtube.com/watch?v=VVvnTByzTmA

LordDragonfang 8y ago
Focussing on "affirmative action" feels uncomfortably close to the exact kind of dog whistling this article criticized up front. As I understand it, most colleges have been using some form of linear regression model on many factors to filter admissions in a way that maximize returns (i.e. alumni donations) for decades now. Surely there is other evidence which shows that if SAT scores were actually a poor heuristic of potential success, schools would have stopped relying on them by now. (without having to bring race into the conversation)
- samatman 8y ago
  He didn't bring race into the conversation; you did. Affirmative action is precisely an adjustment of admissions criteria such that ceterus paribus a candidate is more likely to be admitted. Yes, it's based on race, but if it was based on lottery, it would have the same effect: students with lower SAT scores are admitted to schools which otherwise require a certain threshold. That they don't do as well, shows that the SAT is measuring something relevant to success in education. Note that affirmative action would have this effect even if average IQ were the same across races, because schools vary in the SAT thresholds they set, and this would have the effect of propelling students into settings they aren't prepared to flourish in.

pgeorgi 8y ago

> > Note that IQ tests correlate with SAT scores! IQ tests (of which there are a ton) are regularly calibrated, internationally comparable (there are some reasonably culturally neutral test protocols) and stable while SAT is a US-only metric. Maybe SAT is calibrated to correlate with IQ tests?

barry-cotter 8y ago
The SAT is not calibrated to correlate with IQ tests. Being correlated with IQ tests is an unavoidable consequence of what it is. It’s a high stakes test with limited subject matter meant to measure scholastic aptitude. That will unavoidably be highly g loaded. It’s not culture fair, like Raven’s Matrices, because it assumes you speak English at a native level and have been exposed to a middle school curriculum.
- asabjorn 8y ago
  It sounds fair to assume some basic skills for admittance, such as a completed middle school competency level and native level proficiency in the language advanced university level material will be taught in. Any person without those will be severely disadvantaged relative to the peers that don’t.

hhjinks 8y ago

> we use IQ to select for success We do? I have never seen IQ used for anything but online wankery. None of my most skilled coworkers have struck me as people outside the bell curve, and we have never had our IQ test results requested, or even measured in most cases.

barry-cotter 8y ago
The SAT, SSAT, GRE, MCAT, LSAT and GMAT are all glorified IQ tests. They’re used for entrance to selective colleges and graduate programmes. Lots of people will be quite happy to tell you where they went to school and what they studied which is the non-vulgar way of signalling your IQ. Intelligence:Knowns and Unknowns > are used for many purposes, such as selection, diagnosis, and evaluation. Many of the most widely used tests are not intended to measure intelligence itself but some closely related construct: scholastic aptitude, school achievement, specific abilities, etc. Such tests are es- pecially important for selection purposes. For preparatory school, it's the SSAT; for college, the SAT or ACT; for graduate school, the GRE; for medical school, the MCAT; for law school, the LSAT; for business school, the GMAT. Scores on intelligence-related tests matter, and the stakes can be high. http://differentialclub.wdfiles.com/local--files/definitions-structure-and-measurement/Intelligence-Knowns-and-unknowns.pdf http://differentialclub.wdfiles.com/local--files/definitions...
- gaius 8y ago
  I scored very well on the GMAT and it’s really testing how diligently you prepare for it, and that is the predictor of success, not the content of the test itself, IMHO.
  - barry-cotter 8y ago
    I’m sure you did improve. Practice effects exist. They’re also limited. Just as I will get faster if I train to run the 100m or run a marathon you can get better at the GMAT, LSAT etc. But there’s a plateau you hit where you can’t get better. If you try to take someone from 50th percentile to 99th on the GMAT in the majority of cases you will fail. An increase of 0.26 standard deviations isn’t nothing but on the GMAT it’s only ~30 points. https://digitalcommons.ilr.cornell.edu/cgi/viewcontent.cgi?article=1012&amp=&context=articles&amp=&sei-redir=1&referer=https%253A%252F%252Fscholar.google.com%252Fscholar%253Fhl%253Den%2526as_sdt%253D0%25252C5%2526q%253Dgmat%252Bpractice%252Beffect%2526btnG%253D#search=%22gmat%20practice%20effect%22 https://digitalcommons.ilr.cornell.edu/cgi/viewcontent.cgi?a... Retesting in selection: a meta-analysis of coaching and practice effects for tests of cognitive ability. Previous studies have indicated that as many as 25% to 50% of applicants in organizational and educational settings are retested with measures of cognitive ability. Researchers have shown that practice effects are found across measurement occasions such that scores improve when these applicants retest. In this study, the authors used meta-analysis to summarize the results of 50 studies of practice effects for tests of cognitive ability. Results from 107 samples and 134,436 participants revealed an adjusted overall effect size of 0.26. Moderator analyses indicated that effects were larger when practice was accompanied by test coaching and when identical forms were used. Additional research is needed to understand the impact of retesting on the validity inferences drawn from test scores.

commandlinefan 8y ago

> we use IQ to select for success, then say IQ is correlated with success. We don’t, though. The most financially successful people in the world are sports stars, movie stars, and business owners, none of whose success has anything to do with IQ scores. He bangs on IQ, but nobody actually uses IQ to select for anything – it’s actually prohibited by law in the US (https://www.britannica.com/event/Griggs-v-Duke-Power-Co https://www.britannica.com/event/Griggs-v-Duke-Power-Co). We do use SAT scores to determine admission to college, and SAT scores correlate pretty well with IQ scores, though, so I guess he’s indirectly complaining about SAT scores being used for college admissions? There’s nothing particularly new about this, though – people have complained for eons that measuring people’s test scores only tracks how good they are at taking tests. The problem is, true or not, tests are the only objective way we’ve ever come up with to figure out if somebody actually knows something.