5 ms·

The point Taleb is making is that IQ works well in the lower half of the distribution, but not in the upper half. Which makes sense, considering that questions

by pps43 8y ago

The point Taleb is making is that IQ works well in the lower half of the distribution, but not in the upper half. Which makes sense, considering that questions for IQ tests are created by people with close to average intelligence. Imagine third-graders writing college-level math tests, or beginner chess players coming up with chess problems to separate masters from grandmasters.

chillacy 8y ago
But many of the tests aren’t binary in outcome, like ravens progressive matrices or N back.
- pps43 8y ago
  The problem is not whether the answer is binary. The problem is that the author of the test needs to have at least the same level of intelligence (and probably even higher) than what the test is measuring. Otherwise how can you tell if the answer is correct? So to get a good IQ score one needs to be able to figure out what a test designer (with average IQ) thought the correct answer should be.
  - catskul2 8y ago
    > The problem is that the author of the test needs to have at least the same level of intelligence (and probably even higher) than what the test is measuring. What makes you believe that?
    - pps43 8y ago
      I gave an example above. Imagine you ask second graders to write math tests for college. They might extrapolate from their experience like this "We started working with single-digit numbers, like 2+3 or 4*2. Then we learned to work with two-digit numbers, 23+17 or 17-73. High school math should probably be three-digit numbers, college will be four-digit numbers, and math professor must be able to work with eight-digit numbers." This is actually a reasonable approach (extrapolating from known data), but do you think this test will separate math ability well among college students?
      - slavik81 8y ago
        Your example describes a lack of knowledge, not a lack of intelligence.
        pps43 8y ago
        Yet the same argument can be made for (any specific aspect of) intelligence.
        catskul2 8y ago
        Not really. Many parts of standard intelligence tests are pattern matching. In many cases the patterns can be generated and verified algorithmically and the number of elements as well as time limits are varied. The maker of the test would not necessarily do well on it. In general, detecting and analyzing an ability does not require the ability that is being analyzed.
  - pgeorgi 8y ago
    > Otherwise how can you tell if the answer is correct? For example: By taking much more time and access to all literature while designing the test, which differs from the test protocol. IQ tests aren't quizzes: you don't need to be "clever" to design them, you need to know which aspect of thought process you want to measure. The value comes from their stability: there are tons of tests based on different methodologies and different focus points and still they roughly agree on their ranking, and remain relatively stable for individuals over time.
    - pps43 8y ago
      I gave examples elsewhere in this thread that illustrate why more time won't help. There is a large difference between solving the same type of problem, only larger, fast, and using abstractions at a higher level. Do you really think that speed of doing long division predicts ability to solve differential equations?
      - pgeorgi 8y ago
        > Do you really think that speed of doing long division predicts ability to solve differential equations? I never said that. I also fail to see how "solving differential equations" has anything to do with intelligence (with the caveat that pathologically low mathematical aptitude might limit the ability to learn it at all - one aspect here is that "unable to do division" might be a predictor for that). But I believe that a properly designed "long division test" could be (part of) a predictor for some dimension of intelligence, and that a time component is usually helpful in test situations since it prevents tools and techniques from equalizing the outcome too much (as, given enough time and sufficient command of appropriate techniques, any problem can be solved).
        pps43 8y ago
        I agree that long division measures some dimension of intelligence. But there are so many dimensions that knowing a coordinate in one dimension does not tell you much about coordinates in other dimensions. Taleb's point is that there is correlation, but it's mostly driven by the left side of the distribution. "Unable to do long division" usually means that this person cannot solve differential equations. "Able to do long division" tells you very little about that person's ability to solve differential equations. The speed of solving simple problems does not necessarily predict ability to solve really complex problems well. For example, a chimp named Ayumu easily beats humans in short-term memory tests: https://www.theguardian.com/science/2013/sep/29/chimp-intelligence-aymu-matsuzawa-kyoto https://www.theguardian.com/science/2013/sep/29/chimp-intell...
LordDragonfang 8y ago
>questions for IQ tests are created by people with close to average intelligence Is this really the case? I would expect the opposite to be true, that people with high IQs and specifically organizations like MENSA that select based on IQ would have a vested interest in being involved in that process.
- pps43 8y ago
  Tanya Khovanova on IQ and MENSA: https://blog.tanyakhovanova.com/2008/12/my-iq/ https://blog.tanyakhovanova.com/2008/12/my-iq/
toasterlovin 8y ago
His main problem is that he’s dead wrong.
barry-cotter 8y ago
> Which makes sense, considering that questions for IQ tests are created by people with close to average intelligence. This isn’t true. Designing, testing questions and norming a new IQ test or updating an existing one is very expensive. At minimum you’ll need a Master’s in Psychology, Psychology or Psychometrics to get those jobs. Look at the Educational Testing Service. https://www.ets.org/research/internship-fellowship?WT.ac=research_36065_internship_161116 https://www.ets.org/research/internship-fellowship?WT.ac=res... > The Harold Gulliksen Psychometric Research Fellowship Program > This program seeks graduate students who have completed their doctoral coursework and are at the dissertation stage of their program, working on a dissertation related to statistics, psychometrics, educational/psychological measurement or quantitative methods. Selected Fellows study at their universities during the academic year and conduct a research project under the supervision of an academic mentor and in consultation with an ETS researcher or psychometrician. Selected Fellows are required to participate in the ETS Summer Internship Program.
- pps43 8y ago
  Here's somebody with intelligence clearly above average making the same point: https://blog.tanyakhovanova.com/2008/12/my-iq/ https://blog.tanyakhovanova.com/2008/12/my-iq/

by pps43 8y ago

chillacy 8y ago

But many of the tests aren’t binary in outcome, like ravens progressive matrices or N back.

pps43 8y ago
The problem is not whether the answer is binary. The problem is that the author of the test needs to have at least the same level of intelligence (and probably even higher) than what the test is measuring. Otherwise how can you tell if the answer is correct? So to get a good IQ score one needs to be able to figure out what a test designer (with average IQ) thought the correct answer should be.
- catskul2 8y ago
  > The problem is that the author of the test needs to have at least the same level of intelligence (and probably even higher) than what the test is measuring. What makes you believe that?
  - pps43 8y ago
    I gave an example above. Imagine you ask second graders to write math tests for college. They might extrapolate from their experience like this "We started working with single-digit numbers, like 2+3 or 4*2. Then we learned to work with two-digit numbers, 23+17 or 17-73. High school math should probably be three-digit numbers, college will be four-digit numbers, and math professor must be able to work with eight-digit numbers." This is actually a reasonable approach (extrapolating from known data), but do you think this test will separate math ability well among college students?
    - slavik81 8y ago
      Your example describes a lack of knowledge, not a lack of intelligence.
      - pps43 8y ago
        Yet the same argument can be made for (any specific aspect of) intelligence.
        catskul2 8y ago
        Not really. Many parts of standard intelligence tests are pattern matching. In many cases the patterns can be generated and verified algorithmically and the number of elements as well as time limits are varied. The maker of the test would not necessarily do well on it. In general, detecting and analyzing an ability does not require the ability that is being analyzed.
- pgeorgi 8y ago
  > Otherwise how can you tell if the answer is correct? For example: By taking much more time and access to all literature while designing the test, which differs from the test protocol. IQ tests aren't quizzes: you don't need to be "clever" to design them, you need to know which aspect of thought process you want to measure. The value comes from their stability: there are tons of tests based on different methodologies and different focus points and still they roughly agree on their ranking, and remain relatively stable for individuals over time.
  - pps43 8y ago
    I gave examples elsewhere in this thread that illustrate why more time won't help. There is a large difference between solving the same type of problem, only larger, fast, and using abstractions at a higher level. Do you really think that speed of doing long division predicts ability to solve differential equations?
    - pgeorgi 8y ago
      > Do you really think that speed of doing long division predicts ability to solve differential equations? I never said that. I also fail to see how "solving differential equations" has anything to do with intelligence (with the caveat that pathologically low mathematical aptitude might limit the ability to learn it at all - one aspect here is that "unable to do division" might be a predictor for that). But I believe that a properly designed "long division test" could be (part of) a predictor for some dimension of intelligence, and that a time component is usually helpful in test situations since it prevents tools and techniques from equalizing the outcome too much (as, given enough time and sufficient command of appropriate techniques, any problem can be solved).
      - pps43 8y ago
        I agree that long division measures some dimension of intelligence. But there are so many dimensions that knowing a coordinate in one dimension does not tell you much about coordinates in other dimensions. Taleb's point is that there is correlation, but it's mostly driven by the left side of the distribution. "Unable to do long division" usually means that this person cannot solve differential equations. "Able to do long division" tells you very little about that person's ability to solve differential equations. The speed of solving simple problems does not necessarily predict ability to solve really complex problems well. For example, a chimp named Ayumu easily beats humans in short-term memory tests: https://www.theguardian.com/science/2013/sep/29/chimp-intelligence-aymu-matsuzawa-kyoto https://www.theguardian.com/science/2013/sep/29/chimp-intell...

LordDragonfang 8y ago

>questions for IQ tests are created by people with close to average intelligence Is this really the case? I would expect the opposite to be true, that people with high IQs and specifically organizations like MENSA that select based on IQ would have a vested interest in being involved in that process.

pps43 8y ago
Tanya Khovanova on IQ and MENSA: https://blog.tanyakhovanova.com/2008/12/my-iq/ https://blog.tanyakhovanova.com/2008/12/my-iq/

toasterlovin 8y ago

His main problem is that he’s dead wrong.

barry-cotter 8y ago

> Which makes sense, considering that questions for IQ tests are created by people with close to average intelligence. This isn’t true. Designing, testing questions and norming a new IQ test or updating an existing one is very expensive. At minimum you’ll need a Master’s in Psychology, Psychology or Psychometrics to get those jobs. Look at the Educational Testing Service. https://www.ets.org/research/internship-fellowship?WT.ac=research_36065_internship_161116 https://www.ets.org/research/internship-fellowship?WT.ac=res... > The Harold Gulliksen Psychometric Research Fellowship Program > This program seeks graduate students who have completed their doctoral coursework and are at the dissertation stage of their program, working on a dissertation related to statistics, psychometrics, educational/psychological measurement or quantitative methods. Selected Fellows study at their universities during the academic year and conduct a research project under the supervision of an academic mentor and in consultation with an ETS researcher or psychometrician. Selected Fellows are required to participate in the ETS Summer Internship Program.

pps43 8y ago
Here's somebody with intelligence clearly above average making the same point: https://blog.tanyakhovanova.com/2008/12/my-iq/ https://blog.tanyakhovanova.com/2008/12/my-iq/