8 ms·

IQ tests aren't a great way of testing the intelligence of an LLM, hell they're even controversial when used to measure human intelligence. There is plenty of

IQ tests aren't a great way of testing the intelligence of an LLM, hell they're even controversial when used to measure human intelligence. There is plenty of psych research out there pointing out the potential issues with IQ tests so I won't rehash them here. More importantly, the tests were only ever designed and validated to test human intelligence relative to all other humans that take the test. Unless we're expecting that LLMs (or AIs) have an intelligence that is functionally identical to humans, using an IQ test for them is like expecting a speedometer designed for a bicycle to work for my car on the interstate.

wanderingstan 3y ago
Reminds me of Moravec’s Paradox. Colloquially we still say that someone who wins at the Jeopardy quiz show is “smart”, when of course now any LLM is better than humans at this “test”. Likewise proficiency at chess used to be seen as a sign of intelligence.
aleph_minus_one 3y ago
> There is plenty of psych research out there pointing out the potential issues with IQ tests so I won't rehash them here. I would honestly be interested in some arguments and references on this topic. As far as I am aware, the g factor > https://en.wikipedia.org/wiki/G_factor_(psychometrics) https://en.wikipedia.org/wiki/G_factor_(psychometrics) is one of the best researched psychometric values. The problem with IQ tests to my knowledge is rather that a lot of jobs don't require a lot of (IQ) intelligence; too much intelligence is actually rather a drawback.
- _heimdall 3y ago
  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6927908/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6927908/ Wish I had better links handy that I could share right now, but this is a decent overview with plenty of further references. My wife has the psych degrees so I will probably butcher this trying to explain more deeply, but here goes nothing. The g factor is one concern, though as far as I'm aware its more of a detail than a fundamental problem. There have been complaints related to racism, though in everything I've seen the issue is actually how the IQ results are used rather than a problem with the test itself. My understanding of the more fundamental problem is related to how IQ tests attempt to quantify intelligence as a reliable predictor of future success. The tests ultimately assume that quantitative, rather than qualitative, measures are the right predictor to use. The tests also lean heavily analytical, biasing the test against anyone who is more "right brained". With regards to prediction, its extremely difficult to show any predictive accuracy without a logical loop. IQ tests are already used as screening metrics before one is given some opportunities. There isn't any way that I known of to tease that data apart and see how the person would have succeeded without the IQ test being used. Again, this is definitely a layman's understanding if the issues. I'm not in the psych field so someone here may very well be able to correct me where I've gone off the rails. I know I have listened to plenty of conversations between my wife and colleagues, grad students, etc that convinced me that there are good reason for IQ tests to be in question.
  - aleph_minus_one 3y ago
    Thanks for the paper link. > My understanding of the more fundamental problem is related to how IQ tests attempt to quantify intelligence as a reliable predictor of future success. This was indeed the original motivation in the past. The fact that most variants to define the concept of "intelligence" in some kind of replicable test lead to to the observation that all of these concepts are strongly correlated with the g factor, so in my opinion the evidence is on the side that we formalized "intelligence" mostly correctly. Now that we have a decent measurable concept of intelligence available, we can actually do studies with which other things (such as "future success") intelligence/IQ is correlated. And here the results are in my opinion, well, ... somewhat complicated. For example, as of today, it is quite well-known that if you consider having a steep career as "success", it's rather that being in the dark triad is quite helpful. Another example: it is well-known that attractive people are often thought by other people to be more smart than they actually are (halo effect; https://en.wikipedia.org/wiki/Halo_effect#Role_of_attractiveness https://en.wikipedia.org/wiki/Halo_effect#Role_of_attractive... ; see also https://en.wikipedia.org/wiki/Physical_attractiveness_stereotype https://en.wikipedia.org/wiki/Physical_attractiveness_stereo...). Thus: the problem rather seems to be that intelligence/IQ is not a predictor for quite some things that people would love it to be associated with. But this is in my opinion not a problem with IQ, but with social expectations.
Veuxdo 3y ago
> Unless we're expecting that LLMs (or AIs) have an intelligence that is functionally identical to humans Actually, yes, that's exactly what we should be expecting when being sold "artificial intelligence".
- _heimdall 3y ago
  IQ tests are designed and validated around measuring a fairly tight band of intelligence commonly seen in people though. If an actual AI has yet to match human intelligence, and once it has surpassed us, the test wouldn't be applicable. More importantly, it would be missing any intelligence an AI may develop that isn't extremely similar to what we have evolved. I don't think we can assume that an AI will develop exactly like we will, and I think many assume as much when it comes to whether AI will develop emotions that we would recognize.
  - Veuxdo 3y ago
    But what is non-human (or, really, non-animal) intelligence? I posit that there is no such thing, by definition. Computers have long been better than humans at certain tasks. That isn't "intelligence", though. If an AI can't do IQ tests, then it is per-se not intelligent. Find a different word.
    - _heimdall 3y ago
      So are you defining intelligence by what humans can do? Meaning intelligence is pegged against what we evolved for so anything deviating from that is unintelligent by definition? If so, I can't really argue with that since its purely semantics. I don't know the point of defining intelligence there at that point though, and it risks diverting the moral and ethical concerns of AI development by falling into a semantic debate. We could make a new word for non-human intelligence, but what does that solve?
password54321 3y ago
>hell they're even controversial when used to measure human intelligence You could come up with an IQ test that perfectly tests general intelligence and it would still be controversial. Most don't want a score attached to their level of intelligence.
- canjobear 3y ago
  The people that don’t like IQ tests usually deny that “general intelligence” is a thing at all.
  - _heimdall 3y ago
    I've heard plenty of reasoned arguments over the years (that I would butcher trying to articulate) that raise specific issue with the methodology of the tests, biases towards analytical thinking, and whether the tests are really predictive.

wanderingstan 3y ago

Reminds me of Moravec’s Paradox. Colloquially we still say that someone who wins at the Jeopardy quiz show is “smart”, when of course now any LLM is better than humans at this “test”. Likewise proficiency at chess used to be seen as a sign of intelligence.

aleph_minus_one 3y ago

> There is plenty of psych research out there pointing out the potential issues with IQ tests so I won't rehash them here. I would honestly be interested in some arguments and references on this topic. As far as I am aware, the g factor > https://en.wikipedia.org/wiki/G_factor_(psychometrics) https://en.wikipedia.org/wiki/G_factor_(psychometrics) is one of the best researched psychometric values. The problem with IQ tests to my knowledge is rather that a lot of jobs don't require a lot of (IQ) intelligence; too much intelligence is actually rather a drawback.

_heimdall 3y ago
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6927908/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6927908/ Wish I had better links handy that I could share right now, but this is a decent overview with plenty of further references. My wife has the psych degrees so I will probably butcher this trying to explain more deeply, but here goes nothing. The g factor is one concern, though as far as I'm aware its more of a detail than a fundamental problem. There have been complaints related to racism, though in everything I've seen the issue is actually how the IQ results are used rather than a problem with the test itself. My understanding of the more fundamental problem is related to how IQ tests attempt to quantify intelligence as a reliable predictor of future success. The tests ultimately assume that quantitative, rather than qualitative, measures are the right predictor to use. The tests also lean heavily analytical, biasing the test against anyone who is more "right brained". With regards to prediction, its extremely difficult to show any predictive accuracy without a logical loop. IQ tests are already used as screening metrics before one is given some opportunities. There isn't any way that I known of to tease that data apart and see how the person would have succeeded without the IQ test being used. Again, this is definitely a layman's understanding if the issues. I'm not in the psych field so someone here may very well be able to correct me where I've gone off the rails. I know I have listened to plenty of conversations between my wife and colleagues, grad students, etc that convinced me that there are good reason for IQ tests to be in question.
- aleph_minus_one 3y ago
  Thanks for the paper link. > My understanding of the more fundamental problem is related to how IQ tests attempt to quantify intelligence as a reliable predictor of future success. This was indeed the original motivation in the past. The fact that most variants to define the concept of "intelligence" in some kind of replicable test lead to to the observation that all of these concepts are strongly correlated with the g factor, so in my opinion the evidence is on the side that we formalized "intelligence" mostly correctly. Now that we have a decent measurable concept of intelligence available, we can actually do studies with which other things (such as "future success") intelligence/IQ is correlated. And here the results are in my opinion, well, ... somewhat complicated. For example, as of today, it is quite well-known that if you consider having a steep career as "success", it's rather that being in the dark triad is quite helpful. Another example: it is well-known that attractive people are often thought by other people to be more smart than they actually are (halo effect; https://en.wikipedia.org/wiki/Halo_effect#Role_of_attractiveness https://en.wikipedia.org/wiki/Halo_effect#Role_of_attractive... ; see also https://en.wikipedia.org/wiki/Physical_attractiveness_stereotype https://en.wikipedia.org/wiki/Physical_attractiveness_stereo...). Thus: the problem rather seems to be that intelligence/IQ is not a predictor for quite some things that people would love it to be associated with. But this is in my opinion not a problem with IQ, but with social expectations.

Veuxdo 3y ago

> Unless we're expecting that LLMs (or AIs) have an intelligence that is functionally identical to humans Actually, yes, that's exactly what we should be expecting when being sold "artificial intelligence".

_heimdall 3y ago
IQ tests are designed and validated around measuring a fairly tight band of intelligence commonly seen in people though. If an actual AI has yet to match human intelligence, and once it has surpassed us, the test wouldn't be applicable. More importantly, it would be missing any intelligence an AI may develop that isn't extremely similar to what we have evolved. I don't think we can assume that an AI will develop exactly like we will, and I think many assume as much when it comes to whether AI will develop emotions that we would recognize.
- Veuxdo 3y ago
  But what is non-human (or, really, non-animal) intelligence? I posit that there is no such thing, by definition. Computers have long been better than humans at certain tasks. That isn't "intelligence", though. If an AI can't do IQ tests, then it is per-se not intelligent. Find a different word.
  - _heimdall 3y ago
    So are you defining intelligence by what humans can do? Meaning intelligence is pegged against what we evolved for so anything deviating from that is unintelligent by definition? If so, I can't really argue with that since its purely semantics. I don't know the point of defining intelligence there at that point though, and it risks diverting the moral and ethical concerns of AI development by falling into a semantic debate. We could make a new word for non-human intelligence, but what does that solve?

password54321 3y ago

>hell they're even controversial when used to measure human intelligence You could come up with an IQ test that perfectly tests general intelligence and it would still be controversial. Most don't want a score attached to their level of intelligence.

canjobear 3y ago
The people that don’t like IQ tests usually deny that “general intelligence” is a thing at all.
- _heimdall 3y ago
  I've heard plenty of reasoned arguments over the years (that I would butcher trying to articulate) that raise specific issue with the methodology of the tests, biases towards analytical thinking, and whether the tests are really predictive.