11 ms·

The way human IQ testing developed is that researchers noticed people who excel in one cognitive task tend to do well in others - the “positive manifold.” They

by gpt5 1y ago

The way human IQ testing developed is that researchers noticed people who excel in one cognitive task tend to do well in others - the “positive manifold.” They then hypothesized a general factor, “g,” to explain this pattern. Early tests (e.g., Binet–Simon; later Stanford–Binet and Wechsler) sampled a wide range of tasks, and researchers used correlations and factor analysis to extract the common component, then norm it around 100 with a SD of 15 and call it IQ. IQ tend to meaningfully predicts performance across some domains especially education and work, and shows high test–retest stability from late adolescence through adulthood. It is also tend to be consistent between high quality tests, despite a wide variety of testing methods. It looks like this site just uses human rated public IQ tests. But it would have been more interesting if an IQ test was developed specifically for AI. I.e. a test that would aim to Factor out the strength of a model general cognitive ability across a wide variety of tasks. It is probably doable by doing principal component analysis on a large set of benchmarks available today.

anywhichway 1y ago
One potential issue with that approach is the factors wouldn't stay very constant across generations of AI models. While a lot of people have used various methods to try to gauge the strength of various AI models, one of my favorites is this time horizon analysis [1] which took coding tasks of various lengths and looked at how long it takes to humans to complete those tasks and compared that to chance that the AI would successfully complete the task. Then they looked at various threshholds to see how long of tasks an AI could generally complete with a certain percent threshold. They found the length of a task that AI is able to complete with a various threshholds is doubling about every 7 months. The reason I found this to be an interesting approach is both because AI seems to struggling with coding tasks as the problem grows in complexity and also because being able to give it more complex tasks is an important metric both for coding tasks or more generally just asking AIs to act as independent agents. In my experience increasing the complexity of a problem has a much larger performance falloff for AI than in humans where the task would just take longer, so this approach makes a lot of intuitive sense to me. [1] - https://theaidigest.org/time-horizons https://theaidigest.org/time-horizons
sc077y 1y ago
I believe the ARC-AGI benchmark fits that description, it's sort of an IQ test for LLMs, though I would caution against using the word "Intelligence" for LLMs.
deleted 1y ago
[deleted]
krapp 1y ago
I imagine the value of something like this is for business owners to choose which LLMs they can replace their employees with, so it using human IQ tests is relevant.
- azernik 1y ago
  The point is that the correlation between doing well on these tasks and doing well on other (directly useful) tasks is well established for humans, but not well established for LLMs. If the employees' job is taking IQ tests, then this is a great measure for employers. Otherwise, it doesn't measure anything useful.
  - b112 1y ago
    Otherwise, it doesn't measure anything useful. Oh it measures a useful metric, absolutely, as aspects of an IQ test validate certain types of cognition. Those types of cognition have been found to map to real-world employment of the same. If an AI is so incapable of performing admirably on an IQ test for those types of cognition, then one thing we're certainly measuring is that it's incapable of handling that 'class' of cognition if the conditions change in minuscule and tiny ways. And that's quite important. For example, if the model appears to perform specific work tasks well, related to a class of cognition, then cannot do the same category of cognitive tasks outside of that scope, we're measuring lack of adaptability or true cognitive capability. It's definitely measuring something. Such as, will the model go sideways with small deviations on task or input? That's a nice start.
    - azernik 1y ago
      "Those types of cognition have been found to map to real-world employment of the same." ...in humans. That correlation has not been established for LLMs.
YetAnotherNick 1y ago
ARC-AGI challenge aims for that. In fact the objective is even more strict that the tasks must be trivial for most humans given time.
nsoonhui 1y ago
Another component of this theory concerning g is that it's largely genetic, and immune to "intervention" AKA stability as you mentioned. See the classic "The Bell Curve" for a full exposition. Which makes me wonder what's the point of all the intervention in the form of teaching/parenting styles and whatnot, if g factor is nature and immutable by large? What's the logic of the educators here?
- cedilla 1y ago
  "The Bell Curve" is, let's say, highly controversial and not a good introduction into the topic. Its claim that genetics are the main predictor of IQ, which was very weakly supported at the time, has been completely and undeniably refuted by science in the thirty years since it's publication.
  - brabel 1y ago
    Really? If not genetics then what is it? Just random??
    - smokel 1y ago
      If it isn't nature, then it probably is nurture. Averaged over the entire population, that is indeed mostly random.
    - mdp2021 1y ago
      Personal development. It's a "subtle" skill. You train it (though maybe less directly than other skills).
    - pona-a 1y ago
      Education? Or more directly, socio-economics. The many of the subjects tested never had any experience with this kind of formal testing, had little to no education, and of course predictably failed on several abstract tasks. It might be that the very pattern of sitting down and intensely focusing on apparently meaningless problems isn't as innate as expected.
alphazard 1y ago
IQ is a discovery about how intelligence occurs in humans. As you mentioned, a single factor explains most of the performance of a human on an IQ test, and that model is better than theories of multiple orthogonal intelligences. To contrast, 5 orthogonal factors are the best model we have for human personality. The first question to ask is "do LLMs also have a general factor?". How much of an LLMs performance on an IQ test can be explained by a single positive correlation between all questions? I would expect LLMs to perform much better on memory tasks than anything else, and I wouldn't be surprised if that was holding up their scores. Is there a multi factor model that better explains LLM performance on these tests?
- naveen99 1y ago
  Some points on the 4 ? or 5 dimensional personality space correlate with higher iq though.
  - alphazard 1y ago
    That may be the case. The personality traits are mostly uncorrelated with one another. I was trying to give an example of what a successful multi factor model looks like (the Big 5) to then contrast it with a multi factor model that doesn't work well (theories of multiple intelligences).
- famouswaffles 1y ago
  >The first question to ask is "do LLMs also have a general factor?". Yes, there is some research about it here - https://www.sciencedirect.com/science/article/pii/S0160289624000527 https://www.sciencedirect.com/science/article/pii/S016028962...
technothrasher 1y ago
> The way human IQ testing developed is that researchers noticed people who excel in one cognitive task tend to do well in others My son took an IQ test and it wouldn't score him because he breaks this assumption. He was getting 98% in some tasks and 2% in others. The psychologist giving him the test said it was unlikely enough pattern that they couldn't get an IQ result for him. He's been diagnosed with non-verbal learning disability, and this is apparently common for nvld folks.
- whatshisface 1y ago
  A rule whose exceptions are one in a hundred fails for three million Americans.
  - alistairSH 1y ago
    And works for 290 million plus…
- erikerikson 1y ago
  A term of use for your son is twice exceptional. The GP is correct about the theoretical basis of the tests. Note the use of "tend" in the quote. Even those who fit that better tend to have differential strengths so that has shown to be too simple. Over time the models of intelligence have complected adding EQ (emotional quotient), SQ (social q...), and so on but IQ was first, continues to be considered useful in some ways even as it's also been considered an oppression by some.
  - PaulHoule 1y ago
    I've maxxed out any test or subscale of verbal intelligence that I've taken since I was 12 or so but my schizotaxic brain glitches enough that my problem solving ability in test environments is a bit degraded, still at least an SD above average, but enough that I get an IQ test score that is just high and not off the charts.
  - Workaccount2 1y ago
    EQ, SQ and whatever other-q, are not really a true thing. They're more feel good tests for Facebook dwellers who get confused on IQ tests. There are social assessments, but they are for identifying disorders.
VoodooJuJu 1y ago
[dead]
RcouF1uZ4gsC 1y ago
Good point. There is probably a correlation between how fast a human can do math problems and how intelligent they are in general. But a very trivial python program running on a normal computer will beat the fastest human at math problems in terms of speed. Even though it does nothing else useful
pessimizer 1y ago
> The way human IQ testing developed is that researchers noticed people who excel in one cognitive task tend to do well in others - the “positive manifold.” I'm pretty sure that this is not true, and that the tests were developed to measure children's intellectual development, and whether they were behind or ahead for their age. A bunch of people saw them and decided that it was far better than the primitive tests they had devised in an attempt to limit immigration from southern Europe, or to justify legal discrimination against black people, and wished a universal intelligence scalar into existence. They justify this by saying that the results on this year's test correlate with the results of last years test. They are not laughed at. The thing it most correlates with is the value of your parent's car or cars.
- jcranmer 1y ago
  That's my recollection, too--IQ was developed to judge kids' "mental age" (how far ahead or behind they are compared to their peers in their current grade) and was only later retrofitted onto the g factor model of intelligence.
belter 1y ago
If a model can get an IQ of 120, but can't draw clocks at a precise requested time, or properly count the b's in blueberry, can we then agree IQ tests don't measure intelligence?
- zahlman 1y ago
  No, because that requires the assumption that the model expresses intelligence.
- petesergeant 1y ago
  > or properly count the b's in blueberry This is more akin to you being unable to tell apart the syllables or tones in a very unfamiliar language.
eldenring 1y ago
I don't think something like this works when you can change the model, retrain it, etc. Or at least its much more difficult to do.
rzz3 1y ago
> But it would have been more interesting if an IQ test was developed specifically for AI. Isn’t that basically what the ARC tests are?
- x-complexity 1y ago
  > Isn’t that basically what the ARC tests are? Reductively, yes. IMO, the ARC tests & the visual pattern IQ tests (e.g. Raven's) have little difference, especially if the Raven tests require the taker to draw out the answer. https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices

by gpt5 1y ago

anywhichway 1y ago

One potential issue with that approach is the factors wouldn't stay very constant across generations of AI models. While a lot of people have used various methods to try to gauge the strength of various AI models, one of my favorites is this time horizon analysis [1] which took coding tasks of various lengths and looked at how long it takes to humans to complete those tasks and compared that to chance that the AI would successfully complete the task. Then they looked at various threshholds to see how long of tasks an AI could generally complete with a certain percent threshold. They found the length of a task that AI is able to complete with a various threshholds is doubling about every 7 months. The reason I found this to be an interesting approach is both because AI seems to struggling with coding tasks as the problem grows in complexity and also because being able to give it more complex tasks is an important metric both for coding tasks or more generally just asking AIs to act as independent agents. In my experience increasing the complexity of a problem has a much larger performance falloff for AI than in humans where the task would just take longer, so this approach makes a lot of intuitive sense to me. [1] - https://theaidigest.org/time-horizons https://theaidigest.org/time-horizons

sc077y 1y ago

I believe the ARC-AGI benchmark fits that description, it's sort of an IQ test for LLMs, though I would caution against using the word "Intelligence" for LLMs.

deleted 1y ago

[deleted]

krapp 1y ago

I imagine the value of something like this is for business owners to choose which LLMs they can replace their employees with, so it using human IQ tests is relevant.

azernik 1y ago
The point is that the correlation between doing well on these tasks and doing well on other (directly useful) tasks is well established for humans, but not well established for LLMs. If the employees' job is taking IQ tests, then this is a great measure for employers. Otherwise, it doesn't measure anything useful.
- b112 1y ago
  Otherwise, it doesn't measure anything useful. Oh it measures a useful metric, absolutely, as aspects of an IQ test validate certain types of cognition. Those types of cognition have been found to map to real-world employment of the same. If an AI is so incapable of performing admirably on an IQ test for those types of cognition, then one thing we're certainly measuring is that it's incapable of handling that 'class' of cognition if the conditions change in minuscule and tiny ways. And that's quite important. For example, if the model appears to perform specific work tasks well, related to a class of cognition, then cannot do the same category of cognitive tasks outside of that scope, we're measuring lack of adaptability or true cognitive capability. It's definitely measuring something. Such as, will the model go sideways with small deviations on task or input? That's a nice start.
  - azernik 1y ago
    "Those types of cognition have been found to map to real-world employment of the same." ...in humans. That correlation has not been established for LLMs.

YetAnotherNick 1y ago

ARC-AGI challenge aims for that. In fact the objective is even more strict that the tasks must be trivial for most humans given time.

nsoonhui 1y ago

Another component of this theory concerning g is that it's largely genetic, and immune to "intervention" AKA stability as you mentioned. See the classic "The Bell Curve" for a full exposition. Which makes me wonder what's the point of all the intervention in the form of teaching/parenting styles and whatnot, if g factor is nature and immutable by large? What's the logic of the educators here?

cedilla 1y ago
"The Bell Curve" is, let's say, highly controversial and not a good introduction into the topic. Its claim that genetics are the main predictor of IQ, which was very weakly supported at the time, has been completely and undeniably refuted by science in the thirty years since it's publication.
- brabel 1y ago
  Really? If not genetics then what is it? Just random??
  - smokel 1y ago
    If it isn't nature, then it probably is nurture. Averaged over the entire population, that is indeed mostly random.
  - mdp2021 1y ago
    Personal development. It's a "subtle" skill. You train it (though maybe less directly than other skills).
  - pona-a 1y ago
    Education? Or more directly, socio-economics. The many of the subjects tested never had any experience with this kind of formal testing, had little to no education, and of course predictably failed on several abstract tasks. It might be that the very pattern of sitting down and intensely focusing on apparently meaningless problems isn't as innate as expected.

alphazard 1y ago

IQ is a discovery about how intelligence occurs in humans. As you mentioned, a single factor explains most of the performance of a human on an IQ test, and that model is better than theories of multiple orthogonal intelligences. To contrast, 5 orthogonal factors are the best model we have for human personality. The first question to ask is "do LLMs also have a general factor?". How much of an LLMs performance on an IQ test can be explained by a single positive correlation between all questions? I would expect LLMs to perform much better on memory tasks than anything else, and I wouldn't be surprised if that was holding up their scores. Is there a multi factor model that better explains LLM performance on these tests?

naveen99 1y ago
Some points on the 4 ? or 5 dimensional personality space correlate with higher iq though.
- alphazard 1y ago
  That may be the case. The personality traits are mostly uncorrelated with one another. I was trying to give an example of what a successful multi factor model looks like (the Big 5) to then contrast it with a multi factor model that doesn't work well (theories of multiple intelligences).
famouswaffles 1y ago
>The first question to ask is "do LLMs also have a general factor?". Yes, there is some research about it here - https://www.sciencedirect.com/science/article/pii/S0160289624000527 https://www.sciencedirect.com/science/article/pii/S016028962...

technothrasher 1y ago

> The way human IQ testing developed is that researchers noticed people who excel in one cognitive task tend to do well in others My son took an IQ test and it wouldn't score him because he breaks this assumption. He was getting 98% in some tasks and 2% in others. The psychologist giving him the test said it was unlikely enough pattern that they couldn't get an IQ result for him. He's been diagnosed with non-verbal learning disability, and this is apparently common for nvld folks.

whatshisface 1y ago
A rule whose exceptions are one in a hundred fails for three million Americans.
- alistairSH 1y ago
  And works for 290 million plus…
erikerikson 1y ago
A term of use for your son is twice exceptional. The GP is correct about the theoretical basis of the tests. Note the use of "tend" in the quote. Even those who fit that better tend to have differential strengths so that has shown to be too simple. Over time the models of intelligence have complected adding EQ (emotional quotient), SQ (social q...), and so on but IQ was first, continues to be considered useful in some ways even as it's also been considered an oppression by some.
- PaulHoule 1y ago
  I've maxxed out any test or subscale of verbal intelligence that I've taken since I was 12 or so but my schizotaxic brain glitches enough that my problem solving ability in test environments is a bit degraded, still at least an SD above average, but enough that I get an IQ test score that is just high and not off the charts.
- Workaccount2 1y ago
  EQ, SQ and whatever other-q, are not really a true thing. They're more feel good tests for Facebook dwellers who get confused on IQ tests. There are social assessments, but they are for identifying disorders.

VoodooJuJu 1y ago

[dead]

RcouF1uZ4gsC 1y ago

Good point. There is probably a correlation between how fast a human can do math problems and how intelligent they are in general. But a very trivial python program running on a normal computer will beat the fastest human at math problems in terms of speed. Even though it does nothing else useful

pessimizer 1y ago

> The way human IQ testing developed is that researchers noticed people who excel in one cognitive task tend to do well in others - the “positive manifold.” I'm pretty sure that this is not true, and that the tests were developed to measure children's intellectual development, and whether they were behind or ahead for their age. A bunch of people saw them and decided that it was far better than the primitive tests they had devised in an attempt to limit immigration from southern Europe, or to justify legal discrimination against black people, and wished a universal intelligence scalar into existence. They justify this by saying that the results on this year's test correlate with the results of last years test. They are not laughed at. The thing it most correlates with is the value of your parent's car or cars.

jcranmer 1y ago
That's my recollection, too--IQ was developed to judge kids' "mental age" (how far ahead or behind they are compared to their peers in their current grade) and was only later retrofitted onto the g factor model of intelligence.

belter 1y ago

If a model can get an IQ of 120, but can't draw clocks at a precise requested time, or properly count the b's in blueberry, can we then agree IQ tests don't measure intelligence?

zahlman 1y ago
No, because that requires the assumption that the model expresses intelligence.
petesergeant 1y ago
> or properly count the b's in blueberry This is more akin to you being unable to tell apart the syllables or tones in a very unfamiliar language.

eldenring 1y ago

I don't think something like this works when you can change the model, retrain it, etc. Or at least its much more difficult to do.

rzz3 1y ago

> But it would have been more interesting if an IQ test was developed specifically for AI. Isn’t that basically what the ARC tests are?

x-complexity 1y ago
> Isn’t that basically what the ARC tests are? Reductively, yes. IMO, the ARC tests & the visual pattern IQ tests (e.g. Raven's) have little difference, especially if the Raven tests require the taker to draw out the answer. https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices