3 ms·

IQ tests for models seem...somewhat flawed. For example, most (if not all) IQ tests will test you on working memory. Meaning that you'll be given a string of c

IQ tests for models seem...somewhat flawed. For example, most (if not all) IQ tests will test you on working memory. Meaning that you'll be given a string of characters and numbers, and then you'll have to re-iterate them in some ordered fashion. That is completely trivial for a machine, and will give a large skewed max score. Same with detecting differences. A typical task is to be shown two different pictures, and find the difference between those. Again, totally trivial task for a machine. Or the vocabulary test. Quite trivial for language models. The final IQ score will be some weighted and scaled score that consists of all those different parts. When I took the WAIS-IV, that's how it worked. On the other hand, excluding those (trivial for machine) parts would give a score which may not mirror human intelligence, as far as scoring/testing goes.

TacticalCoder 3y ago
> IQ tests for models seem...somewhat flawed. They are for all the reasons you explain. Here's a wonderful paper (which I probably found here on HN) on the subject: On the Measure of Intelligence, by Francois Chollet (working for Google), from 2019: https://arxiv.org/abs/1911.01547 https://arxiv.org/abs/1911.01547 He comes up with many tests which are relatively simple for humans that AI models cannot solve. If you don't want to read all the paper, jump to the figures (starting from page 48). The goal, of course, is not to train an AI on the correct results of the exact problems in that paper: that'd just be more memory testing. Once Claude solves that, things are going to get really wild!
singularity2001 3y ago
> That is completely trivial for a machine If given as arbitrary images the tests become a bit more 'fair' It will be fascinating to see the race between the invention of new tests which humans excel at and the progress of models. ... until neither us nor the machines can think of any more categorically different tests

TacticalCoder 3y ago

> IQ tests for models seem...somewhat flawed. They are for all the reasons you explain. Here's a wonderful paper (which I probably found here on HN) on the subject: On the Measure of Intelligence, by Francois Chollet (working for Google), from 2019: https://arxiv.org/abs/1911.01547 https://arxiv.org/abs/1911.01547 He comes up with many tests which are relatively simple for humans that AI models cannot solve. If you don't want to read all the paper, jump to the figures (starting from page 48). The goal, of course, is not to train an AI on the correct results of the exact problems in that paper: that'd just be more memory testing. Once Claude solves that, things are going to get really wild!

singularity2001 3y ago

> That is completely trivial for a machine If given as arbitrary images the tests become a bit more 'fair' It will be fascinating to see the race between the invention of new tests which humans excel at and the progress of models. ... until neither us nor the machines can think of any more categorically different tests