6 ms·

I'm quite skeptical of analyses like this one, because I doubt the metrics themselves. Emergence is something that is intuitively noticed by human observers. Th

by p-e-w 4y ago

I'm quite skeptical of analyses like this one, because I doubt the metrics themselves. Emergence is something that is intuitively noticed by human observers. The desire to quantify everything then leads to the creation of (imperfect) metrics designed to capture what the observers already know. Those same metrics are then taken as the definition of the properties said to be emergent, and articles like this one are among the consequences of that choice. The paper's claim is essentially "these metrics which appear to demonstrate emergence can be replaced by other metrics that also represent model behavior, but that do not have scale discontinuities, so emergence isn't a real phenomenon". But an equally valid interpretation would be "none of these metrics actually capture the properties we are truly interested in". Which, given the complexity of what we are dealing with here, seems entirely reasonable. It's not like we suddenly learned how to accurately quantify performance at language tasks. The whole reason LLMs are so great in the first place is because traditional 'mechanical' language models suck so bad.

amw-zero 4y ago
You just described science, and why you don't believe in science.
- p-e-w 4y ago
  If by "don't believe in science" you mean "don't believe that every metric claimed to be representing a phenomenon actually does represent that phenomenon", you are correct.
Centigonal 4y ago
I think the claim that "these metrics which appear to demonstrate emergence can be replaced by other metrics that also represent model behavior, but that do not have scale discontinuities (emergence)" is really powerful. That might mean that behaviors we consider emergent are the consequence of a process that scales continuously with model size. i.e.: there may exist a bijection between, say, a step-function `can_do_arithmetic(size)` and a smooth, continuous function `arithmetic_skill_metric(size)` If we can use continuous metrics to back out the step-function equivalents, that'll help us predict when and how to get particular abilities to "emerge." For example: If a change results in a steeper slope on the continuous metric, we can predict it would cause the associated capability to emerge at relatively smaller model sizes.
- p-e-w 4y ago
  > That might mean that behaviors we consider emergent are the consequence of a process that scales continuously with model size. Or it might mean that the metrics used are worthless for describing high-level model behavior. That's my whole point. Emergent behavior was observed, which is why we want those metrics so we can try to understand what is going on. But just because we have metrics that exhibit discontinuities at scales where humans observe emergence, doesn't mean that those metrics really represent the hard-to-define behavioral changes we have observed.
  - Centigonal 4y ago
    I'm not sure I understand. I think you're suggesting that the metrics currently being used to assess emergence of new capabilities in LLMs are imperfect and potentially worthless, but I don't understand what is missing. Using the example of 4 digit multiplication in the source paper: The researcher wants to know if the model has developed the ability to multiply two four-digit integers, so they generate a battery of such problems, e.g. "what is 4363*1285? output only your answer." The metric is what percentage of the problems the LLM answers correctly. This is pretty much the same way a human observer would identify the same emergent behavior, and also how we assess it in other humans. It's not some contrived metric that's detached from the emergent ability in question.
    - agalunar 4y ago
      That's not at all how I would assess whether someone understood how to do multiplication – I would ask them to explain their process for performing multiplication or at least to show their work. Maybe I'm misunderstanding what you're saying, though?
      - Centigonal 4y ago
        If you wanted to know if someone can multiply two numbers, the simplest and quickest way is probably to ask them to multiply two numbers and see if they get it right. Remember that, in this case, all I want to know is if they have that capability or not. Also, I don't care whether they can tell me how multiplication works or not, only if they can multiply two arbitrary numbers. (here, the abstraction to people breaks down, because people can walk you through their reasoning - whereas for an LLM, explaining how multiplication works and performing multiplication are very different tasks) There are weaknesses to this approach: you get a binary yes/no, and you have no idea how close they got. You can't tell if they just made a small math error or if they don't even know what numbers are. Going back to the LLM setting, this is why a continuous metric is useful, compared to one that experiences step-function behavior.
TheOnly92 4y ago
I agree that perhaps the metrics are not as useful themselves, but I think you're giving too little credit to the paper where maybe some credit is due. I think the paper is correct that there are no "emergent abilities", i.e. abilities that might suddenly appear when scale of the model is increased. And though it might not be accurate, but the paper did make some effort to formalize and I think it is a good attempt to kind of prove the point. However as we recognize, there are still some weird discontinuities in which at one point the model is useless and suddenly it becomes very useful. This "discontinuity" IMHO is probably just perceptional, but the underlying metric is continuous.
beders 4y ago
> Emergence is something that is intuitively noticed by human observers. If you can't quantify that, then it is just a hunch. What exactly do you think "observers already know"?
- p-e-w 4y ago
  There are many fundamental qualities that can be observed but not quantified. General intelligence being one of them (intelligence tests measure some aspects of it, but not others). If everything that is real could be quantified, we wouldn't need AI. Traditional computing is already absolutely phenomenal at dealing with quantifiable systems. The whole point of wanting "artificial intelligence" is because we don't know how to quantify the high-level properties of speech, thought, intellect, and consciousness. And not for lack of trying.
freehorse 4y ago
> Emergence is something that is intuitively noticed by human observers. The takeaway from this paper is actually that this says more about the human observers than the LLMs themselves. Humans observe "emergence" using some loose metrics themselves. Whether we formalise the metrics in a quantitative way or we stay in the realm of humnan intuition is not important; we still use some criteria to analyse things, in the latter case these are more clear and amenable to critique, in the former case they are obscure and can evade critique by moving goalposts and ground easily. In the end of the day, either we talk about quantification or just qualitative observations, it is the same phenomenon we observe with the same qualities. The problem is that human intuition uses a lot of discontinuous metrics; we judge if an LLM passes or fails what we ask for it, but it is harder to judge the underlying tokenisation process as itself. For this reason, and considering the findings of this paper, the observations and claims of emergence in LLMs carry less value now, imo.
- p-e-w 4y ago
  Calling human intuition "just another metric" ignores the fact that human intuition performs spectacularly well at many high-level tasks. Whatever "insight" and "understanding" actually mean, there is no denying that they are immensely useful, which is why we want AIs that can replicate them. When trying to understand complex systems that don't yield to quantitative analysis in an obvious way, the starting point should be to assume that the intuitive evaluation is (roughly) correct, until proven otherwise. Trying to cast this intuition into a simple metric and then using that metric (or other simple metrics) to demonstrate that the intuition is wrong is circular reasoning.
  - kelseyfrog 4y ago
    > human intuition performs spectacularly well at many high-level tasks. It's also the number one most common cognitive bias. Humans are especially prone to reification - the confusion that the construction of a measure equates to an objective reality. Humans are often launder subjectivity through the creation of metrics without 1) knowing they've done so, and 2) become emotional when accused of having done so.
JohnFen 4y ago
> Emergence is something that is intuitively noticed by human observers. The problem, of course, is that people's intuition is particularly awful for this sort of thing. We have a very strong tendency to anthropomorphize everything, and that illusion can be quite overpowering.
roguecoder 3y ago
Lots of things are intuitively noticed by human observers, only some of which exist. We are heuristic pattern-finding machines who believe in ESP and fairies: why wouldn't we also find mythology in the machines we build?

by p-e-w 4y ago

amw-zero 4y ago

You just described science, and why you don't believe in science.

p-e-w 4y ago
If by "don't believe in science" you mean "don't believe that every metric claimed to be representing a phenomenon actually does represent that phenomenon", you are correct.

Centigonal 4y ago

I think the claim that "these metrics which appear to demonstrate emergence can be replaced by other metrics that also represent model behavior, but that do not have scale discontinuities (emergence)" is really powerful. That might mean that behaviors we consider emergent are the consequence of a process that scales continuously with model size. i.e.: there may exist a bijection between, say, a step-function `can_do_arithmetic(size)` and a smooth, continuous function `arithmetic_skill_metric(size)` If we can use continuous metrics to back out the step-function equivalents, that'll help us predict when and how to get particular abilities to "emerge." For example: If a change results in a steeper slope on the continuous metric, we can predict it would cause the associated capability to emerge at relatively smaller model sizes.

p-e-w 4y ago
> That might mean that behaviors we consider emergent are the consequence of a process that scales continuously with model size. Or it might mean that the metrics used are worthless for describing high-level model behavior. That's my whole point. Emergent behavior was observed, which is why we want those metrics so we can try to understand what is going on. But just because we have metrics that exhibit discontinuities at scales where humans observe emergence, doesn't mean that those metrics really represent the hard-to-define behavioral changes we have observed.
- Centigonal 4y ago
  I'm not sure I understand. I think you're suggesting that the metrics currently being used to assess emergence of new capabilities in LLMs are imperfect and potentially worthless, but I don't understand what is missing. Using the example of 4 digit multiplication in the source paper: The researcher wants to know if the model has developed the ability to multiply two four-digit integers, so they generate a battery of such problems, e.g. "what is 4363*1285? output only your answer." The metric is what percentage of the problems the LLM answers correctly. This is pretty much the same way a human observer would identify the same emergent behavior, and also how we assess it in other humans. It's not some contrived metric that's detached from the emergent ability in question.
  - agalunar 4y ago
    That's not at all how I would assess whether someone understood how to do multiplication – I would ask them to explain their process for performing multiplication or at least to show their work. Maybe I'm misunderstanding what you're saying, though?
    - Centigonal 4y ago
      If you wanted to know if someone can multiply two numbers, the simplest and quickest way is probably to ask them to multiply two numbers and see if they get it right. Remember that, in this case, all I want to know is if they have that capability or not. Also, I don't care whether they can tell me how multiplication works or not, only if they can multiply two arbitrary numbers. (here, the abstraction to people breaks down, because people can walk you through their reasoning - whereas for an LLM, explaining how multiplication works and performing multiplication are very different tasks) There are weaknesses to this approach: you get a binary yes/no, and you have no idea how close they got. You can't tell if they just made a small math error or if they don't even know what numbers are. Going back to the LLM setting, this is why a continuous metric is useful, compared to one that experiences step-function behavior.

TheOnly92 4y ago

I agree that perhaps the metrics are not as useful themselves, but I think you're giving too little credit to the paper where maybe some credit is due. I think the paper is correct that there are no "emergent abilities", i.e. abilities that might suddenly appear when scale of the model is increased. And though it might not be accurate, but the paper did make some effort to formalize and I think it is a good attempt to kind of prove the point. However as we recognize, there are still some weird discontinuities in which at one point the model is useless and suddenly it becomes very useful. This "discontinuity" IMHO is probably just perceptional, but the underlying metric is continuous.

beders 4y ago

> Emergence is something that is intuitively noticed by human observers. If you can't quantify that, then it is just a hunch. What exactly do you think "observers already know"?

p-e-w 4y ago
There are many fundamental qualities that can be observed but not quantified. General intelligence being one of them (intelligence tests measure some aspects of it, but not others). If everything that is real could be quantified, we wouldn't need AI. Traditional computing is already absolutely phenomenal at dealing with quantifiable systems. The whole point of wanting "artificial intelligence" is because we don't know how to quantify the high-level properties of speech, thought, intellect, and consciousness. And not for lack of trying.

freehorse 4y ago

> Emergence is something that is intuitively noticed by human observers. The takeaway from this paper is actually that this says more about the human observers than the LLMs themselves. Humans observe "emergence" using some loose metrics themselves. Whether we formalise the metrics in a quantitative way or we stay in the realm of humnan intuition is not important; we still use some criteria to analyse things, in the latter case these are more clear and amenable to critique, in the former case they are obscure and can evade critique by moving goalposts and ground easily. In the end of the day, either we talk about quantification or just qualitative observations, it is the same phenomenon we observe with the same qualities. The problem is that human intuition uses a lot of discontinuous metrics; we judge if an LLM passes or fails what we ask for it, but it is harder to judge the underlying tokenisation process as itself. For this reason, and considering the findings of this paper, the observations and claims of emergence in LLMs carry less value now, imo.

p-e-w 4y ago
Calling human intuition "just another metric" ignores the fact that human intuition performs spectacularly well at many high-level tasks. Whatever "insight" and "understanding" actually mean, there is no denying that they are immensely useful, which is why we want AIs that can replicate them. When trying to understand complex systems that don't yield to quantitative analysis in an obvious way, the starting point should be to assume that the intuitive evaluation is (roughly) correct, until proven otherwise. Trying to cast this intuition into a simple metric and then using that metric (or other simple metrics) to demonstrate that the intuition is wrong is circular reasoning.
- kelseyfrog 4y ago
  > human intuition performs spectacularly well at many high-level tasks. It's also the number one most common cognitive bias. Humans are especially prone to reification - the confusion that the construction of a measure equates to an objective reality. Humans are often launder subjectivity through the creation of metrics without 1) knowing they've done so, and 2) become emotional when accused of having done so.

JohnFen 4y ago

> Emergence is something that is intuitively noticed by human observers. The problem, of course, is that people's intuition is particularly awful for this sort of thing. We have a very strong tendency to anthropomorphize everything, and that illusion can be quite overpowering.

roguecoder 3y ago

Lots of things are intuitively noticed by human observers, only some of which exist. We are heuristic pattern-finding machines who believe in ESP and fairies: why wouldn't we also find mythology in the machines we build?