5 ms·

I think the claim that "these metrics which appear to demonstrate emergence can be replaced by other metrics that also represent model behavior, but that do not

I think the claim that "these metrics which appear to demonstrate emergence can be replaced by other metrics that also represent model behavior, but that do not have scale discontinuities (emergence)" is really powerful. That might mean that behaviors we consider emergent are the consequence of a process that scales continuously with model size. i.e.: there may exist a bijection between, say, a step-function `can_do_arithmetic(size)` and a smooth, continuous function `arithmetic_skill_metric(size)` If we can use continuous metrics to back out the step-function equivalents, that'll help us predict when and how to get particular abilities to "emerge." For example: If a change results in a steeper slope on the continuous metric, we can predict it would cause the associated capability to emerge at relatively smaller model sizes.

p-e-w 3y ago
> That might mean that behaviors we consider emergent are the consequence of a process that scales continuously with model size. Or it might mean that the metrics used are worthless for describing high-level model behavior. That's my whole point. Emergent behavior was observed, which is why we want those metrics so we can try to understand what is going on. But just because we have metrics that exhibit discontinuities at scales where humans observe emergence, doesn't mean that those metrics really represent the hard-to-define behavioral changes we have observed.
- Centigonal 3y ago
  I'm not sure I understand. I think you're suggesting that the metrics currently being used to assess emergence of new capabilities in LLMs are imperfect and potentially worthless, but I don't understand what is missing. Using the example of 4 digit multiplication in the source paper: The researcher wants to know if the model has developed the ability to multiply two four-digit integers, so they generate a battery of such problems, e.g. "what is 4363*1285? output only your answer." The metric is what percentage of the problems the LLM answers correctly. This is pretty much the same way a human observer would identify the same emergent behavior, and also how we assess it in other humans. It's not some contrived metric that's detached from the emergent ability in question.
  - agalunar 3y ago
    That's not at all how I would assess whether someone understood how to do multiplication – I would ask them to explain their process for performing multiplication or at least to show their work. Maybe I'm misunderstanding what you're saying, though?
    - Centigonal 3y ago
      If you wanted to know if someone can multiply two numbers, the simplest and quickest way is probably to ask them to multiply two numbers and see if they get it right. Remember that, in this case, all I want to know is if they have that capability or not. Also, I don't care whether they can tell me how multiplication works or not, only if they can multiply two arbitrary numbers. (here, the abstraction to people breaks down, because people can walk you through their reasoning - whereas for an LLM, explaining how multiplication works and performing multiplication are very different tasks) There are weaknesses to this approach: you get a binary yes/no, and you have no idea how close they got. You can't tell if they just made a small math error or if they don't even know what numbers are. Going back to the LLM setting, this is why a continuous metric is useful, compared to one that experiences step-function behavior.
      - agalunar 3y ago
        > Remember that, in this case, all I want to know is if they have that capability or not. Also, I don't care whether they can tell me how multiplication works or not, only if they can multiply two arbitrary numbers. Right, but unless you know something about their process, the only way you can determine whether they have the ability to (correctly) multiply two arbitrary four-digit numbers is to have them demonstrate multiplying every combination of four-digit numbers. One can easily imagine a system that gets most answers correct but fails certain cases (e.g. only carrying between pairs of digits). The reason why an inexhaustive test (using only several dozen examples) works to some degree with schoolchildren is because we know something about their method: the algorithm for multiplication they've been exposed to – that they're explicitly being taught – is an algorithm that we know to be correct. That has a very large effect on the likelihood of different failure modes and the evidence required to have a particular degree of confidence. Furthermore, for "system 2" activities (to borrow the term from Kahneman), we can reasonably expect a person's description of their process to match the process they actually performed. (There are exceptions: people will produced incorrect post hoc explanations for their behavior under sufficient duress when "I don't know" isn't perceived as an acceptable answer.) But I'm personally not aware of any reason to believe this about LLMs. I don't know why the network's actual process for performing multiplication should have anything to do with the text it produces after the fact when asked to explain its work. So I'm not debating that a reasonable way to guess whether a person can (correctly) multiply two numbers is to "ask them to multiply two numbers and see if they get it right"; I'm disagreeing that this works with LLMs.

p-e-w 3y ago

> That might mean that behaviors we consider emergent are the consequence of a process that scales continuously with model size. Or it might mean that the metrics used are worthless for describing high-level model behavior. That's my whole point. Emergent behavior was observed, which is why we want those metrics so we can try to understand what is going on. But just because we have metrics that exhibit discontinuities at scales where humans observe emergence, doesn't mean that those metrics really represent the hard-to-define behavioral changes we have observed.

Centigonal 3y ago
I'm not sure I understand. I think you're suggesting that the metrics currently being used to assess emergence of new capabilities in LLMs are imperfect and potentially worthless, but I don't understand what is missing. Using the example of 4 digit multiplication in the source paper: The researcher wants to know if the model has developed the ability to multiply two four-digit integers, so they generate a battery of such problems, e.g. "what is 4363*1285? output only your answer." The metric is what percentage of the problems the LLM answers correctly. This is pretty much the same way a human observer would identify the same emergent behavior, and also how we assess it in other humans. It's not some contrived metric that's detached from the emergent ability in question.
- agalunar 3y ago
  That's not at all how I would assess whether someone understood how to do multiplication – I would ask them to explain their process for performing multiplication or at least to show their work. Maybe I'm misunderstanding what you're saying, though?
  - Centigonal 3y ago
    If you wanted to know if someone can multiply two numbers, the simplest and quickest way is probably to ask them to multiply two numbers and see if they get it right. Remember that, in this case, all I want to know is if they have that capability or not. Also, I don't care whether they can tell me how multiplication works or not, only if they can multiply two arbitrary numbers. (here, the abstraction to people breaks down, because people can walk you through their reasoning - whereas for an LLM, explaining how multiplication works and performing multiplication are very different tasks) There are weaknesses to this approach: you get a binary yes/no, and you have no idea how close they got. You can't tell if they just made a small math error or if they don't even know what numbers are. Going back to the LLM setting, this is why a continuous metric is useful, compared to one that experiences step-function behavior.
    - agalunar 3y ago
      > Remember that, in this case, all I want to know is if they have that capability or not. Also, I don't care whether they can tell me how multiplication works or not, only if they can multiply two arbitrary numbers. Right, but unless you know something about their process, the only way you can determine whether they have the ability to (correctly) multiply two arbitrary four-digit numbers is to have them demonstrate multiplying every combination of four-digit numbers. One can easily imagine a system that gets most answers correct but fails certain cases (e.g. only carrying between pairs of digits). The reason why an inexhaustive test (using only several dozen examples) works to some degree with schoolchildren is because we know something about their method: the algorithm for multiplication they've been exposed to – that they're explicitly being taught – is an algorithm that we know to be correct. That has a very large effect on the likelihood of different failure modes and the evidence required to have a particular degree of confidence. Furthermore, for "system 2" activities (to borrow the term from Kahneman), we can reasonably expect a person's description of their process to match the process they actually performed. (There are exceptions: people will produced incorrect post hoc explanations for their behavior under sufficient duress when "I don't know" isn't perceived as an acceptable answer.) But I'm personally not aware of any reason to believe this about LLMs. I don't know why the network's actual process for performing multiplication should have anything to do with the text it produces after the fact when asked to explain its work. So I'm not debating that a reasonable way to guess whether a person can (correctly) multiply two numbers is to "ask them to multiply two numbers and see if they get it right"; I'm disagreeing that this works with LLMs.