14 ms·

Characterizing emergent phenomena in large language models

77 points by fofoz 4y ago

djoldman 4y ago
Paper: https://openreview.net/forum?id=yzkSU5zdwD https://openreview.net/forum?id=yzkSU5zdwD
xpe 4y ago
Wikipedia has a fine definition of what _emergent_ means: > In philosophy, systems theory, science, and art, emergence occurs when an entity is observed to have properties its parts do not have on their own, properties or behaviors that emerge only when the parts interact in a wider whole. The linked article uses this definition: > we discuss the phenomena of emergent abilities, which we define as abilities that are not present in small models but are present in larger models The concept in the paper has to do with capabilities / abilities that grow non-linearly as a function of model size. This is distinctly different from _emergent behavior_ in systems theory. <opinion>The authors and reviewers could find a better word for their concept. There is no need to muddle the concept.</opinion> Furthermore, the idea that networks of certain sizes are necessary for certain kinds of representational abilities is not new. Perhaps a term exists already?
- xpe 4y ago
  This comment says it more eloquently than I did: https://news.ycombinator.com/item?id=34051845 https://news.ycombinator.com/item?id=34051845
mjburgess 4y ago
> we discuss the phenomena of emergent abilities, which we define as abilities that are not present in small models but are present in larger models Reading anything by major researchers in AI feels like an adversarial battle where they're trying to misuse as much technical scientific and philosophical language as possible and we adjacent are trying to hold the line. In philosophy and esp. the philosophy of science, emergence is a relation between a whole and its parts such that a property of the whole does not obtain just in virtue of properties of its parts taken in isolation. "Emergence" has this prior positive, semi-magical, scientific association which confuses the issue in this case. No properties of the LLM obtain from its parts differently as parameters scale, the mechanism is the same. The performance differs not due to emergence, but due to the "modelling gap" present between the statistical structure of free text and that of mathematics. With enough examples, the gap closes... indeed, you can model the addition function (add(x, y) = x + y) just by an infinite sample of its domains. A better technical term here might be "scale-dependent capabilities". For LLM, simple arithmetic is extremely scale dependent, whereas basic text generation is less-so. The reason for this seems obvious, as given above... so the use of the term "emergence" here I interpert as more PRish mystification.
- xpe 4y ago
  Well said. How did the authors and reviewers miss this?
  - mjburgess 4y ago
    The transition from useless to useful ML models no doubt often seems magical to researchers. But it follows just from the distribution of the training data and from the degree of its compression by the function approximation algorithm they're using. What's "magical" is not their system, but rather that the vast library of text they use for training has useful properties which can be approximated. What researchers are observing is more like the illusion of a "phase transition" in the quality of approximations. This illusion arises because we have discontinuous standards for the approximations. Ie., when assessing free text prediction by LLMs there's very very many ways for them to generate an acceptable answer. For mathematics, there's only one acceptable way. If we applied the same standard/goal to both, no such apparent "quality transition" would occur. LLMs would be exposed as equally good, or equally bad, at prediction regardless of scale.
    - xpe 4y ago
      Interesting arguments. It seems plausible and insightful. IMO, your analysis here deserves a longer write-up. Is it something you are working on?
      - mjburgess 4y ago
        Any person with a "scientific attitude" in this field would find it incredibly easy to observer that the training target for natural language is, f(Q) = {A1..An} -- n being very very large and the target for mathematics is, g(Q) = A1 And the model they're using approximates with, m(Q) = A_guess So it's incredibly easy to model f with m because A_guess just has to be close to one of A1...n; and its very hard to model mathematics because it has to be only A1. The reason articles like this are written isn't because people dont know this; it's because they just do not have a sceptical attitude. And that's a problem I can't fix. If they'd approached this issue with the goal of "finding nothing surprising about this behaviour", ie., trying to make it maximally consistent with existing (basic, trivial, widely-taught) theory, they'd reach this conclusion in under 5min. The problem is their goal is always to find something surprising so they can release some PR about it. There's nothing I can write to fix this problem, it's edemic in the whole field. It makes ML/AI research much more like psychology than physics. Alas!
seydor 4y ago
The X axis here is the training Flops but what about parameter size and how does it account for the different architectures. Comparing apples to shoelaces may not be a fruitful approach or indicative of what to expect from ever-expanding scale. Also , is it emergence or overfitting
evrimoztamur 4y ago
Has there been any efforts in processing calculation prompts, where instead of letting it internally 'compute', it's trained to identify equations and process them with an external calculator instead (perhaps one which outputs not only the result but the individual steps too)?
- obiefernandez 4y ago
  Been wondering the same
- vutekst 4y ago
  Yes: https://twitter.com/goodside/status/1581805503897735168 https://twitter.com/goodside/status/1581805503897735168
- visarga 4y ago
  Language models with toys. The calculator, Python REPL, search engine, database, simulation, games, other AI's can easily blend with large language models lifting some weight off their shoulders. For example, for a physics question the LM could write a small simulation, run the simulation and interpret results back to the user. That's possible when models can do code execution.
ttctciyf 4y ago
There's a quite accessible IAS presentation[1] from another Google researcher on Solving Quantitative Reasoning Problems with Language Models which gives some likely related background on having language models solve this type of math problem, including the "chain of thought" technique mentioned here. I found it pretty interesting and as something of an ML skeptic was a bit surprised at the degree of coherence shown in "reasoning" examples similar to the ones in the linked article. 1: https://www.youtube.com/watch?v=qV4Ku5L4BuMt https://www.youtube.com/watch?v=qV4Ku5L4BuMt
CGamesPlay 4y ago
Do these scale-dependent (I like this adjective better than "emergent") properties survive model distillation? It may be that our training/optimization processes are inefficient and require these scales to achieve, but the underlying model may not actually require the number of parameters that we are giving them. I haven't read any of the papers about distillation yet, does anyone know if this has been tested?
- visarga 4y ago
  Good question, my guess is that you can't distill chain-of-thought or zero shot prompting in small models, they got to be 15-20B parameters or larger. Maybe someone has a link to a related paper?
  - lossolo 4y ago
    For all models smaller than 62B, direct prompting outperforms CoT. The first model where CoT outperforms direct prompting is Flan-cont-PaLM 62B on BBH. For 540B models, there are more settings where CoT outperforms direct prompting, but not all of them. Also, the number can be smaller than 540B. In Suzgun et. al. 2022, the authors show that the 175B InstructGPT and 175B Codex also have better CoT performance than direct prompting. Combining all the results, we get the two numbers 62B and 175B. So yes indeed, to enter the game of scale you do need a ticket to larger models than average. However, there are also other large models like OPT, BLOOM, and the first version of GPT-3. They all have 175B, yet their CoT performance is significantly worse, or even cannot do CoT. source: https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Models-Emergent-Abilities-493876b55df5479d80686f68a1abd72f https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Mo...
    - visarga 4y ago
      Found a paper myself: > Teaching Small Language Models to Reason https://arxiv.org/abs/2212.08410 https://arxiv.org/abs/2212.08410
      - CGamesPlay 4y ago

77 points by fofoz 4y ago

Paper: https://openreview.net/forum?id=yzkSU5zdwD https://openreview.net/forum?id=yzkSU5zdwD

xpe 4y ago

Wikipedia has a fine definition of what _emergent_ means: > In philosophy, systems theory, science, and art, emergence occurs when an entity is observed to have properties its parts do not have on their own, properties or behaviors that emerge only when the parts interact in a wider whole. The linked article uses this definition: > we discuss the phenomena of emergent abilities, which we define as abilities that are not present in small models but are present in larger models The concept in the paper has to do with capabilities / abilities that grow non-linearly as a function of model size. This is distinctly different from _emergent behavior_ in systems theory. <opinion>The authors and reviewers could find a better word for their concept. There is no need to muddle the concept.</opinion> Furthermore, the idea that networks of certain sizes are necessary for certain kinds of representational abilities is not new. Perhaps a term exists already?

xpe 4y ago
This comment says it more eloquently than I did: https://news.ycombinator.com/item?id=34051845 https://news.ycombinator.com/item?id=34051845

mjburgess 4y ago

> we discuss the phenomena of emergent abilities, which we define as abilities that are not present in small models but are present in larger models Reading anything by major researchers in AI feels like an adversarial battle where they're trying to misuse as much technical scientific and philosophical language as possible and we adjacent are trying to hold the line. In philosophy and esp. the philosophy of science, emergence is a relation between a whole and its parts such that a property of the whole does not obtain just in virtue of properties of its parts taken in isolation. "Emergence" has this prior positive, semi-magical, scientific association which confuses the issue in this case. No properties of the LLM obtain from its parts differently as parameters scale, the mechanism is the same. The performance differs not due to emergence, but due to the "modelling gap" present between the statistical structure of free text and that of mathematics. With enough examples, the gap closes... indeed, you can model the addition function (add(x, y) = x + y) just by an infinite sample of its domains. A better technical term here might be "scale-dependent capabilities". For LLM, simple arithmetic is extremely scale dependent, whereas basic text generation is less-so. The reason for this seems obvious, as given above... so the use of the term "emergence" here I interpert as more PRish mystification.

xpe 4y ago
Well said. How did the authors and reviewers miss this?
- mjburgess 4y ago
  The transition from useless to useful ML models no doubt often seems magical to researchers. But it follows just from the distribution of the training data and from the degree of its compression by the function approximation algorithm they're using. What's "magical" is not their system, but rather that the vast library of text they use for training has useful properties which can be approximated. What researchers are observing is more like the illusion of a "phase transition" in the quality of approximations. This illusion arises because we have discontinuous standards for the approximations. Ie., when assessing free text prediction by LLMs there's very very many ways for them to generate an acceptable answer. For mathematics, there's only one acceptable way. If we applied the same standard/goal to both, no such apparent "quality transition" would occur. LLMs would be exposed as equally good, or equally bad, at prediction regardless of scale.
  - xpe 4y ago
    Interesting arguments. It seems plausible and insightful. IMO, your analysis here deserves a longer write-up. Is it something you are working on?
    - mjburgess 4y ago
      Any person with a "scientific attitude" in this field would find it incredibly easy to observer that the training target for natural language is, f(Q) = {A1..An} -- n being very very large and the target for mathematics is, g(Q) = A1 And the model they're using approximates with, m(Q) = A_guess So it's incredibly easy to model f with m because A_guess just has to be close to one of A1...n; and its very hard to model mathematics because it has to be only A1. The reason articles like this are written isn't because people dont know this; it's because they just do not have a sceptical attitude. And that's a problem I can't fix. If they'd approached this issue with the goal of "finding nothing surprising about this behaviour", ie., trying to make it maximally consistent with existing (basic, trivial, widely-taught) theory, they'd reach this conclusion in under 5min. The problem is their goal is always to find something surprising so they can release some PR about it. There's nothing I can write to fix this problem, it's edemic in the whole field. It makes ML/AI research much more like psychology than physics. Alas!

seydor 4y ago

The X axis here is the training Flops but what about parameter size and how does it account for the different architectures. Comparing apples to shoelaces may not be a fruitful approach or indicative of what to expect from ever-expanding scale. Also , is it emergence or overfitting

evrimoztamur 4y ago

Has there been any efforts in processing calculation prompts, where instead of letting it internally 'compute', it's trained to identify equations and process them with an external calculator instead (perhaps one which outputs not only the result but the individual steps too)?

obiefernandez 4y ago
Been wondering the same
vutekst 4y ago
Yes: https://twitter.com/goodside/status/1581805503897735168 https://twitter.com/goodside/status/1581805503897735168
visarga 4y ago
Language models with toys. The calculator, Python REPL, search engine, database, simulation, games, other AI's can easily blend with large language models lifting some weight off their shoulders. For example, for a physics question the LM could write a small simulation, run the simulation and interpret results back to the user. That's possible when models can do code execution.

ttctciyf 4y ago

There's a quite accessible IAS presentation[1] from another Google researcher on Solving Quantitative Reasoning Problems with Language Models which gives some likely related background on having language models solve this type of math problem, including the "chain of thought" technique mentioned here. I found it pretty interesting and as something of an ML skeptic was a bit surprised at the degree of coherence shown in "reasoning" examples similar to the ones in the linked article. 1: https://www.youtube.com/watch?v=qV4Ku5L4BuMt https://www.youtube.com/watch?v=qV4Ku5L4BuMt

CGamesPlay 4y ago

Do these scale-dependent (I like this adjective better than "emergent") properties survive model distillation? It may be that our training/optimization processes are inefficient and require these scales to achieve, but the underlying model may not actually require the number of parameters that we are giving them. I haven't read any of the papers about distillation yet, does anyone know if this has been tested?

visarga 4y ago
Good question, my guess is that you can't distill chain-of-thought or zero shot prompting in small models, they got to be 15-20B parameters or larger. Maybe someone has a link to a related paper?
- lossolo 4y ago
  For all models smaller than 62B, direct prompting outperforms CoT. The first model where CoT outperforms direct prompting is Flan-cont-PaLM 62B on BBH. For 540B models, there are more settings where CoT outperforms direct prompting, but not all of them. Also, the number can be smaller than 540B. In Suzgun et. al. 2022, the authors show that the 175B InstructGPT and 175B Codex also have better CoT performance than direct prompting. Combining all the results, we get the two numbers 62B and 175B. So yes indeed, to enter the game of scale you do need a ticket to larger models than average. However, there are also other large models like OPT, BLOOM, and the first version of GPT-3. They all have 175B, yet their CoT performance is significantly worse, or even cannot do CoT. source: https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Models-Emergent-Abilities-493876b55df5479d80686f68a1abd72f https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Mo...
  - visarga 4y ago
    Found a paper myself: > Teaching Small Language Models to Reason https://arxiv.org/abs/2212.08410 https://arxiv.org/abs/2212.08410
    - CGamesPlay 4y ago