8 ms·

"Intelligence" itself is very ill-defined and we've never been able to measure it properly, IQ is rife with issues. At some point, you just have to be pragmati

"Intelligence" itself is very ill-defined and we've never been able to measure it properly, IQ is rife with issues. At some point, you just have to be pragmatic and measure the questions you want the AI to be good at answering, rather than trying to measure intelligence in general. In that sense, I see this as one more benchmark that collects questions that we want/expect AI to be good at, is not good at yet and have been underrepresented at previous benchmarks. That's obviously valuable, there's nothing "magical" about it. Although it is reasonable to be annoyed at the "Humanity's Last Exam" naming, of course they must have missed plenty of edge-cases like everyone else and it is very arrogant to claim it will be the "Last" one.

esotericimpl 2y ago
This is always the answer for anyone who thinks LLMs are capable of "intelligence". It's good at answering questions that its trained on, I would suggest general intelligence are things you didnt want/train the AI to be good at answering.
- fooker 2y ago
  Are you good at answering questions you are not trained to answer? How about a middle school test in a language you don’t speak?
  - xanderlewis 2y ago
    Yes — reasonably so, anyway. I don't have to have seen millions of prior examples of exactly the same kind in order to tackle a novel problem in mathematics, say.
    - oersted 2y ago
      Well, LLMs are also remarkably good at generalizing. Look at the datasets, they don't literally train on every conceivable type of question the user might ask, the LLM can adapt just as you can. The actual challenge towards general intelligence is that LLMs struggle with certain types of questions even if you *do* train it on millions of examples of that type of question. Mostly questions that require complex logical reasoning, although consistent progress is being done in this direction.
      - godelski 2y ago
        > Well, LLMs are also remarkably good at generalizing. Look at the datasets, they don't literally train on every conceivable type of question the user might ask, the LLM can adapt just as you can. Proof needed. I'm serious. We don't have the datasets. But we do know the size of the datasets. And the sizes suggest incredible amounts of information. Take an estimate of 100 tokens ~= 75 words[0]. What is a trillion tokens? Well, that's 750bn words. There are approximately 450 words on a page[1]. So that's 1.66... bn pages! If we put that in 500 page books, that would come out to 3.33... million books! Llama 3 has a pretraining size of 15T tokens[2] (this does not include training, so more info added later). So that comes to ~50m books. Then, keep in mind that this data is filtered and deduplicated. Even considering a high failure rate in deduplication, this an unimaginable amount of information. [0] https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them https://help.openai.com/en/articles/4936856-what-are-tokens-... [1] https://wordcounter.net/words-per-page https://wordcounter.net/words-per-page [2] https://ai.meta.com/blog/meta-llama-3/ https://ai.meta.com/blog/meta-llama-3/
        oersted 2y ago
        That’s a very good point. I just speak from my experience of fine-tuning pre-trained models. At least at that stage they can memorize new knowledge, that couldn’t have been in the training data, just by seeing it once during fine-tuning (one epoch), which seems magical. Most instruction-tuning datasets are also remarkably small (very roughly <100K samples). This is only possible if the model has internalized the knowledge quite deeply and generally, such that new knowledge is a tiny gradient update on top of existing expectations. But yes I see what you mean, they are dumping practically the whole internet at it, it’s not unreasonable to think that it has memorized a massive proportion of common question types the user might come up with, such that minimal generalization is needed.
        godelski 2y ago
        > that couldn’t have been in the training data I'm curious, how do you know this? I'm not doubting, but is it falsifiable? I also am not going to claim that LLMs only perform recall. They fit functions in a continuous manner. Even if the data is discrete. So they can do more. The question is more about how much more. Another important point is that out of distribution doesn't mean "not in training". This is sometimes conflated, but if it were true then that's a test set lol. OOD means not belonging to the same distribution. Though that's a bit complicated, especially when dealing with high dimensional data
  - godelski 2y ago
    > Are you good at answering questions you are not trained to answer? Yes. Most schooling is designed around this. Pick a random math textbook. Any will do. Read a chapter. Then move to the homework problems. The typical fashion is that the first few problems are quite similar to the examples in the chapter. Often solvable by substitution and repetition. Middle problems generally require a bit of extrapolation. To connect concepts from previous chapters or courses in ways that likely were not explicitly discussed. This has many forms and frequently includes taking the abstract form to practical (i.e. a word problem). Challenge problems are those that require you to extrapolate the information into new domains. Requiring the connection of many ideas and having to filter information for what is useful and not. > How about a middle school test in a language you don’t speak? A language course often makes this explicitly clear. You are trained to learn the rules of the language. Conjugation is a good example. By learning the structure you can hear new words that you've never heard before and extract information about it even if not exactly. There's a reason you don't just learn vocabulary. It's also assumed that by learning vocabulary you'll naturally learn rules. Language is a great example in general. We constantly invent new words. It really is not uncommon for someone you know to be be talking to you and in that discussion drop a word they made up on the spot or just make a sound or a gesture. An entirely novel thing yet you will likely understand. Often this is zero-shot (sometimes it might just appear to be zero-shot but actually isn't)
  - jhbadger 2y ago
    For a while I was into a trivia program on my phone. It was kind of easy, so I decided to set the language to Catalan, a language which I never studied. I was still able to do well, because I could figure out the questions more or less from languages I do know and could generalize from them. It would be interesting to know if you could say, train an LLM on examples from Romance languages but specifically exclude Catalan and see if it could do the same.
  - gjm11 2y ago
    Well ... https://puzzling.stackexchange.com/questions/94326/a-cryptic-crossword-in-indonesian-language-tts-kriptik-bahasa-indonesia https://puzzling.stackexchange.com/questions/94326/a-cryptic... (Someone made a cryptic crossword[1] whose clues and solutions were in the Bahasa Indonesia language, and it was solved by a couple of people who don't speak that language at all.) [1] These are mostly a UK thing; the crosswords in US newspapers are generally of a different type. In a cryptic crossword, each word is given a clue that typically consists of a definition and some wordplay; there are a whole lot of conventions governing the wordplay. So e.g. the clue "Chooses to smash pots (4)" would lead to the answer OPTS; "chooses" is the definition, "smash pots" is the wordplay, wherein "smash" indicates that what follows should be anagrammed (smashed up). Disclaimer #1: it took those people a lot more work than it would have taken them to solve an English-language cryptic crossword of similar difficulty, and they needed a bunch of external resources. (Dis)claimer #2: one of those people was me. Disclaimer #3: I do not claim that something needs to be able to do this sort of thing in order to be called intelligent. Plenty of intelligent people (including plenty of people more intelligent than me) would also be unable to do it.
  - LorenPechtel 2y ago
    That's called innovation, something the current AIs aren't capable of.
visarga 2y ago
> "Intelligence" itself is very ill-defined and we've never been able to measure it properly, IQ is rife with issues. Yes, because it is 1st person exclusively. If you expand a bit, consider "search efficiency". It's no longer just 1st person, it can be social. And it doesn't hide the search space. Intelligence is partially undefined because it doesn't specify the problem space, it is left blank. But "search efficiency" is more scientific and concrete.
JohnMakin 2y ago
> IQ is rife with issues Indeed, and yet people are obsessed with the it and the idea of measuring their own intelligence - I completely do not understand it. I am in an extremely high percentile, but I am a total moron in a lot of areas and if you met me would likely think so as well. It's a poor predictor for just about everything except how good a person is at recognizing patterns (I know there are many different kinds of tests, but inevitably, it feels like this) and how quickly they can reason. But people are obsessed with it (Go on quora and search "IQ", you probably won't half to though, since half the questions there are seemingly about IQ). A thing I like to say is you didn't earn your intelligence any more than a 7'0" man earned his height - to some degree it seems innate (we don't even really know how). This all said, it seems even more pointless to try to "IQ" test an AI in this manner. What does it predict? What is it measuring? And you're not going to be able to use the same questions for more than 1 test, because the AI will "learn" the answers.
- godelski 2y ago
  The lowest IQ thing you can do is be obsessed with IQ. There are known knowns, there are known unknowns, and there are unknown unknowns. The wise man knows he cannot know what he does not know and that it'd be naive to presume he knows when he cannot know how much he doesn't know. Therefore, only the unintelligent man really _knows_ anything.
  - dingnuts 2y ago
    IQ is compute speed, not storage. It has nothing to do with knowledge. IBM used to give one out as part of their hiring process, years ago, and even I took it the entire test was a timed multiple choice exam where every question was looking at an object made out of cubes and choosing the correct orientation of the object from the choices, after the object was arbitrarily rotated according to instructions in the question. Then, IQ can be derived by determining how quickly all participants can answer the questionnaire correctly, and ranking their speeds, and then normalizing the values so 100 is in the middle. Turns out, scores will fall along a bell curve if you do that. You can call that phenomenon whatever, but most people call it IQ and hopefully I've explained well why that has nothing at all to do with static knowledge in this comment.
godelski 2y ago
> "Intelligence" itself is very ill-defined While this is true, it is well agreed upon (by domain experts) that intelligence is distinct from knowledge recall. But that's what most of these tests... test. If you look at IQ tests you'll see that they are attempts to test things that aren't knowledge based. You'll also notice that the main critiques of IQ tests are about how they often actually measure knowledge and that there's bias in natural knowledge acquisition. So even the disagreements about the definition of intelligence make clear that knowledge and intelligence are distinct. I feel that often people conflate "intelligence is ill-defined" with "intelligence has no definition." These two are not in opposition. Being ill-defined is more like "I know I left my phone in the house, but I'm not sure where." This is entirely different from "I lost my phone, it is somewhere in California" or "It is somewhere on Earth" and clearly different from "I lost my phone. I'm unsure if I had a phone. What even is a phone?"
- oersted 2y ago
  Yes agreed, there is indeed a rough consensus on what intelligence is and reasonable ways to approximately measure it. These standard tests have been applied to LLMs from the beginning, they have not proven to be the most helpful to guide research, but there's value to applying benchmarks that have been battle-tested with humans. It's just that OP was questioning this group's criteria for selecting the questions that determine intelligence. Then we get into endless discussions of semantics. At the end of the day, you are just testing which questions your AI performs well on, and you can describe how you chose those questions. Claiming it measures "general intelligence" is just unhelpful and frustrating.
  - godelski 2y ago
    They were applied in the beginning because we really weren't that good at solving the tasks. So like any good researchers, we break it down. But this is like trying to test an elephant but you can't get access to an elephant so you instead train a dog. But putting a dog in an elephant costume doesn't make it an elephant. Sure, dog training will likely mean you can learn to train an elephant faster had you not first trained a dog. Some things transfer, but others don't I also want to stress that there is a rough consensus. But the ML field (which I'm a part of) often ignores this. I'm not sure why. We should be leveraging the work of others, not trying to start from scratch (unless there's good reason, in which case we must be explicit. But I'm just seeing simple claims of "intelligence is ill-defined" and treating that as if that means no definition instead of fuzzy definition. Which gets extra weird when people talk about moving goal posts. That's how progress works? Especially when exploring into the unknown?)

esotericimpl 2y ago

This is always the answer for anyone who thinks LLMs are capable of "intelligence". It's good at answering questions that its trained on, I would suggest general intelligence are things you didnt want/train the AI to be good at answering.

fooker 2y ago
Are you good at answering questions you are not trained to answer? How about a middle school test in a language you don’t speak?
- xanderlewis 2y ago
  Yes — reasonably so, anyway. I don't have to have seen millions of prior examples of exactly the same kind in order to tackle a novel problem in mathematics, say.
  - oersted 2y ago
    Well, LLMs are also remarkably good at generalizing. Look at the datasets, they don't literally train on every conceivable type of question the user might ask, the LLM can adapt just as you can. The actual challenge towards general intelligence is that LLMs struggle with certain types of questions even if you *do* train it on millions of examples of that type of question. Mostly questions that require complex logical reasoning, although consistent progress is being done in this direction.
    - godelski 2y ago
      > Well, LLMs are also remarkably good at generalizing. Look at the datasets, they don't literally train on every conceivable type of question the user might ask, the LLM can adapt just as you can. Proof needed. I'm serious. We don't have the datasets. But we do know the size of the datasets. And the sizes suggest incredible amounts of information. Take an estimate of 100 tokens ~= 75 words[0]. What is a trillion tokens? Well, that's 750bn words. There are approximately 450 words on a page[1]. So that's 1.66... bn pages! If we put that in 500 page books, that would come out to 3.33... million books! Llama 3 has a pretraining size of 15T tokens[2] (this does not include training, so more info added later). So that comes to ~50m books. Then, keep in mind that this data is filtered and deduplicated. Even considering a high failure rate in deduplication, this an unimaginable amount of information. [0] https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them https://help.openai.com/en/articles/4936856-what-are-tokens-... [1] https://wordcounter.net/words-per-page https://wordcounter.net/words-per-page [2] https://ai.meta.com/blog/meta-llama-3/ https://ai.meta.com/blog/meta-llama-3/
      - oersted 2y ago
        That’s a very good point. I just speak from my experience of fine-tuning pre-trained models. At least at that stage they can memorize new knowledge, that couldn’t have been in the training data, just by seeing it once during fine-tuning (one epoch), which seems magical. Most instruction-tuning datasets are also remarkably small (very roughly <100K samples). This is only possible if the model has internalized the knowledge quite deeply and generally, such that new knowledge is a tiny gradient update on top of existing expectations. But yes I see what you mean, they are dumping practically the whole internet at it, it’s not unreasonable to think that it has memorized a massive proportion of common question types the user might come up with, such that minimal generalization is needed.
        godelski 2y ago
        > that couldn’t have been in the training data I'm curious, how do you know this? I'm not doubting, but is it falsifiable? I also am not going to claim that LLMs only perform recall. They fit functions in a continuous manner. Even if the data is discrete. So they can do more. The question is more about how much more. Another important point is that out of distribution doesn't mean "not in training". This is sometimes conflated, but if it were true then that's a test set lol. OOD means not belonging to the same distribution. Though that's a bit complicated, especially when dealing with high dimensional data
- godelski 2y ago
  > Are you good at answering questions you are not trained to answer? Yes. Most schooling is designed around this. Pick a random math textbook. Any will do. Read a chapter. Then move to the homework problems. The typical fashion is that the first few problems are quite similar to the examples in the chapter. Often solvable by substitution and repetition. Middle problems generally require a bit of extrapolation. To connect concepts from previous chapters or courses in ways that likely were not explicitly discussed. This has many forms and frequently includes taking the abstract form to practical (i.e. a word problem). Challenge problems are those that require you to extrapolate the information into new domains. Requiring the connection of many ideas and having to filter information for what is useful and not. > How about a middle school test in a language you don’t speak? A language course often makes this explicitly clear. You are trained to learn the rules of the language. Conjugation is a good example. By learning the structure you can hear new words that you've never heard before and extract information about it even if not exactly. There's a reason you don't just learn vocabulary. It's also assumed that by learning vocabulary you'll naturally learn rules. Language is a great example in general. We constantly invent new words. It really is not uncommon for someone you know to be be talking to you and in that discussion drop a word they made up on the spot or just make a sound or a gesture. An entirely novel thing yet you will likely understand. Often this is zero-shot (sometimes it might just appear to be zero-shot but actually isn't)
- jhbadger 2y ago
  For a while I was into a trivia program on my phone. It was kind of easy, so I decided to set the language to Catalan, a language which I never studied. I was still able to do well, because I could figure out the questions more or less from languages I do know and could generalize from them. It would be interesting to know if you could say, train an LLM on examples from Romance languages but specifically exclude Catalan and see if it could do the same.
- gjm11 2y ago
  Well ... https://puzzling.stackexchange.com/questions/94326/a-cryptic-crossword-in-indonesian-language-tts-kriptik-bahasa-indonesia https://puzzling.stackexchange.com/questions/94326/a-cryptic... (Someone made a cryptic crossword[1] whose clues and solutions were in the Bahasa Indonesia language, and it was solved by a couple of people who don't speak that language at all.) [1] These are mostly a UK thing; the crosswords in US newspapers are generally of a different type. In a cryptic crossword, each word is given a clue that typically consists of a definition and some wordplay; there are a whole lot of conventions governing the wordplay. So e.g. the clue "Chooses to smash pots (4)" would lead to the answer OPTS; "chooses" is the definition, "smash pots" is the wordplay, wherein "smash" indicates that what follows should be anagrammed (smashed up). Disclaimer #1: it took those people a lot more work than it would have taken them to solve an English-language cryptic crossword of similar difficulty, and they needed a bunch of external resources. (Dis)claimer #2: one of those people was me. Disclaimer #3: I do not claim that something needs to be able to do this sort of thing in order to be called intelligent. Plenty of intelligent people (including plenty of people more intelligent than me) would also be unable to do it.
- LorenPechtel 2y ago
  That's called innovation, something the current AIs aren't capable of.

visarga 2y ago

> "Intelligence" itself is very ill-defined and we've never been able to measure it properly, IQ is rife with issues. Yes, because it is 1st person exclusively. If you expand a bit, consider "search efficiency". It's no longer just 1st person, it can be social. And it doesn't hide the search space. Intelligence is partially undefined because it doesn't specify the problem space, it is left blank. But "search efficiency" is more scientific and concrete.

JohnMakin 2y ago

> IQ is rife with issues Indeed, and yet people are obsessed with the it and the idea of measuring their own intelligence - I completely do not understand it. I am in an extremely high percentile, but I am a total moron in a lot of areas and if you met me would likely think so as well. It's a poor predictor for just about everything except how good a person is at recognizing patterns (I know there are many different kinds of tests, but inevitably, it feels like this) and how quickly they can reason. But people are obsessed with it (Go on quora and search "IQ", you probably won't half to though, since half the questions there are seemingly about IQ). A thing I like to say is you didn't earn your intelligence any more than a 7'0" man earned his height - to some degree it seems innate (we don't even really know how). This all said, it seems even more pointless to try to "IQ" test an AI in this manner. What does it predict? What is it measuring? And you're not going to be able to use the same questions for more than 1 test, because the AI will "learn" the answers.

godelski 2y ago
The lowest IQ thing you can do is be obsessed with IQ. There are known knowns, there are known unknowns, and there are unknown unknowns. The wise man knows he cannot know what he does not know and that it'd be naive to presume he knows when he cannot know how much he doesn't know. Therefore, only the unintelligent man really _knows_ anything.
- dingnuts 2y ago
  IQ is compute speed, not storage. It has nothing to do with knowledge. IBM used to give one out as part of their hiring process, years ago, and even I took it the entire test was a timed multiple choice exam where every question was looking at an object made out of cubes and choosing the correct orientation of the object from the choices, after the object was arbitrarily rotated according to instructions in the question. Then, IQ can be derived by determining how quickly all participants can answer the questionnaire correctly, and ranking their speeds, and then normalizing the values so 100 is in the middle. Turns out, scores will fall along a bell curve if you do that. You can call that phenomenon whatever, but most people call it IQ and hopefully I've explained well why that has nothing at all to do with static knowledge in this comment.

godelski 2y ago

> "Intelligence" itself is very ill-defined While this is true, it is well agreed upon (by domain experts) that intelligence is distinct from knowledge recall. But that's what most of these tests... test. If you look at IQ tests you'll see that they are attempts to test things that aren't knowledge based. You'll also notice that the main critiques of IQ tests are about how they often actually measure knowledge and that there's bias in natural knowledge acquisition. So even the disagreements about the definition of intelligence make clear that knowledge and intelligence are distinct. I feel that often people conflate "intelligence is ill-defined" with "intelligence has no definition." These two are not in opposition. Being ill-defined is more like "I know I left my phone in the house, but I'm not sure where." This is entirely different from "I lost my phone, it is somewhere in California" or "It is somewhere on Earth" and clearly different from "I lost my phone. I'm unsure if I had a phone. What even is a phone?"

oersted 2y ago
Yes agreed, there is indeed a rough consensus on what intelligence is and reasonable ways to approximately measure it. These standard tests have been applied to LLMs from the beginning, they have not proven to be the most helpful to guide research, but there's value to applying benchmarks that have been battle-tested with humans. It's just that OP was questioning this group's criteria for selecting the questions that determine intelligence. Then we get into endless discussions of semantics. At the end of the day, you are just testing which questions your AI performs well on, and you can describe how you chose those questions. Claiming it measures "general intelligence" is just unhelpful and frustrating.
- godelski 2y ago
  They were applied in the beginning because we really weren't that good at solving the tasks. So like any good researchers, we break it down. But this is like trying to test an elephant but you can't get access to an elephant so you instead train a dog. But putting a dog in an elephant costume doesn't make it an elephant. Sure, dog training will likely mean you can learn to train an elephant faster had you not first trained a dog. Some things transfer, but others don't I also want to stress that there is a rough consensus. But the ML field (which I'm a part of) often ignores this. I'm not sure why. We should be leveraging the work of others, not trying to start from scratch (unless there's good reason, in which case we must be explicit. But I'm just seeing simple claims of "intelligence is ill-defined" and treating that as if that means no definition instead of fuzzy definition. Which gets extra weird when people talk about moving goal posts. That's how progress works? Especially when exploring into the unknown?)