7 ms·

> we discuss the phenomena of emergent abilities, which we define as abilities that are not present in small models but are present in larger models Reading an

> we discuss the phenomena of emergent abilities, which we define as abilities that are not present in small models but are present in larger models Reading anything by major researchers in AI feels like an adversarial battle where they're trying to misuse as much technical scientific and philosophical language as possible and we adjacent are trying to hold the line. In philosophy and esp. the philosophy of science, emergence is a relation between a whole and its parts such that a property of the whole does not obtain just in virtue of properties of its parts taken in isolation. "Emergence" has this prior positive, semi-magical, scientific association which confuses the issue in this case. No properties of the LLM obtain from its parts differently as parameters scale, the mechanism is the same. The performance differs not due to emergence, but due to the "modelling gap" present between the statistical structure of free text and that of mathematics. With enough examples, the gap closes... indeed, you can model the addition function (add(x, y) = x + y) just by an infinite sample of its domains. A better technical term here might be "scale-dependent capabilities". For LLM, simple arithmetic is extremely scale dependent, whereas basic text generation is less-so. The reason for this seems obvious, as given above... so the use of the term "emergence" here I interpert as more PRish mystification.

xpe 4y ago
Well said. How did the authors and reviewers miss this?
- mjburgess 4y ago
  The transition from useless to useful ML models no doubt often seems magical to researchers. But it follows just from the distribution of the training data and from the degree of its compression by the function approximation algorithm they're using. What's "magical" is not their system, but rather that the vast library of text they use for training has useful properties which can be approximated. What researchers are observing is more like the illusion of a "phase transition" in the quality of approximations. This illusion arises because we have discontinuous standards for the approximations. Ie., when assessing free text prediction by LLMs there's very very many ways for them to generate an acceptable answer. For mathematics, there's only one acceptable way. If we applied the same standard/goal to both, no such apparent "quality transition" would occur. LLMs would be exposed as equally good, or equally bad, at prediction regardless of scale.
  - xpe 4y ago
    Interesting arguments. It seems plausible and insightful. IMO, your analysis here deserves a longer write-up. Is it something you are working on?
    - mjburgess 4y ago
      Any person with a "scientific attitude" in this field would find it incredibly easy to observer that the training target for natural language is, f(Q) = {A1..An} -- n being very very large and the target for mathematics is, g(Q) = A1 And the model they're using approximates with, m(Q) = A_guess So it's incredibly easy to model f with m because A_guess just has to be close to one of A1...n; and its very hard to model mathematics because it has to be only A1. The reason articles like this are written isn't because people dont know this; it's because they just do not have a sceptical attitude. And that's a problem I can't fix. If they'd approached this issue with the goal of "finding nothing surprising about this behaviour", ie., trying to make it maximally consistent with existing (basic, trivial, widely-taught) theory, they'd reach this conclusion in under 5min. The problem is their goal is always to find something surprising so they can release some PR about it. There's nothing I can write to fix this problem, it's edemic in the whole field. It makes ML/AI research much more like psychology than physics. Alas!
- baandang 4y ago
  From Introduction To The Theory Of Complex Systems: • Complex systems can exhibit a rich phase structure and have a huge variety of macrostates that often cannot be inferred from the properties of the elements. This is sometimes referred to as emergence. This is the term and as common a term in complex systems as there is. "scale-dependent capabilities" implies inference from elements. I think some people just don't like the very idea even though it is not unlike the concept of stochastic process. I would think the same reasons to not like the concept of emergence applies to stochastic process. Murray Gell-Mann though couldn't even raise complex systems above arguing if emergence is magical thinking so it is probably a lost cause. Such an interesting field that always ends up as this conversation.
  - naasking 4y ago
    > "scale-dependent capabilities" implies inference from elements. Don't confuse the post-hoc explanation with an a priori inference. We can post-hoc explain water's emergent liquidity property using modern quantum theories, but that doesn't mean we could have inferred it if given Schrodinger's equation and the atomic structure of hydrogen and oxygen. "Scale-dependent capability" is a post-hoc explanation that of course looks obvious in hindsight, just like liquidity and pressure looks obvious in hindsight once you understand electromagnetism and atomic theory.
PartiallyTyped 4y ago
> that a property of the whole does not obtain just in virtue of properties of its parts taken in isolation Thank you. I have been expressing variants of this for a while. A paper that comes to mind is OpenAI's hide and seek. They claim that cooperation is emergent behaviour, but each agent is playing it's own version of prisoner's dilemma, and thus learn to cooperate.
- visarga 4y ago
  That model was not learning from language, it was learning from a simulation. When you can use a simulation to produce training data it is possible to have a model discover new abilities all on its own, like AlphaGo.
seydor 4y ago
Eh eh . imagine how neuroscientists feel
- mjburgess 4y ago
  Well, of late, NS's have been fond of misusing "hallucinate" likewise which means a non-veridical perception. And they're using it to mean a veridical constructed perception. Leading everyone down a path of ever-more mysticism. It would be nice if neuroscientists spoke out against both their own mystical PR and that of AI, but I don't hear it much.
  - oidar 4y ago
    I think the word confabulate would be a better match for what ChatGPT does. When people confabulate they are VERY confident about their invented retellings. This matches the attitude that ChatGPT comes across as when it makes up shit.
naasking 4y ago
I think "scale-dependent capability" is a much more precise term for what they're describing, but I'm not sure that that term doesn't fall under the general umbrella of emergent properties. The opening paragraphs of emergent properties in philosophy [1] cites a number of examples that I would argue are comparable to LLM suddenly becoming able to do arithmetic past a certain scale. > In philosophy and esp. the philosophy of science, emergence is a relation between a whole and its parts such that a property of the whole does not obtain just in virtue of properties of its parts taken in isolation. This is not a settled definition. I think everyone can agree that this applies epistemologically, where studying the parts in isolation cannot always yield sufficient information to predict macroscopic properties, but to claim that all properties of the whole do not reduce to the properties of its parts is controversial. For instance, it seems unlikely that we would have predicted H2O's dipole moment and the phenomenon of surface tension just from studying hydrogen atoms and oxygen atoms in isolation, but it would be incorrect to say that surface tension is not the result of the properties of hydrogen and oxygen in isolation. We simply cannot discover all the relevant properties without studying them together. Edit: to clarify, H2O's dipole moment seems obvious in hindsight when we have a good model for what's going on, and analogously, LLM's ability to do arithmetic seems obvious as a scale-dependent property in hindsight, but that doesn't mean it was obvious that this would happen before it was created. [1] https://plato.stanford.edu/entries/properties-emergent/ https://plato.stanford.edu/entries/properties-emergent/
- mjburgess 4y ago
  Well it's scale dependence is an illusion. It's uniformly able to model Q->SingleAnswer problems, and uniformly able to model Q-> ManyAnswer problems. Basic arithmetic is the former kind, and it becomes useful to people at large scales, ie., accurate on basic arithmetic. This dual condition of "useful-to-people" is the thing introducing the illusion, since it changes depending on what we're modelling. The system isnt acquring any new property. Consider a researcher putting a book on a thin ice-sheet, and then putting a car on it. Here, they're concluding the ice has different properties in each case -- but it doesnt.
  - naasking 4y ago
fumeux_fume 4y ago
Is is that big of a deal? The authors explain their definition of emergent abilities at the beginning of the paper.
- mjburgess 4y ago
  It's a mystification of what's going on -- the term makes it harder to understand, not easier. It's prone to popular misunderstanding, and it seems even to confuse the researchers themselves.
YeGoblynQueenne 4y ago
Thanks for bringing some sense into the debate. It's scandalising to see how the machine learning research community is so ready to jump on to such ... innovative uses of terminology. Take "few shot learning", for instance. OpenAI's preprint introducing GPT-3 was titled "Large Language Models are Few Shot Learners" [1]. This was promptly adopted by the community, even though LLMs need first to be trained on millions of examples before they can accept "few", and they don't even learn from those few (because no weight updates). So it's not really "few shot" and it's not really "learning", and yet, here we are. "Large Language Models are few-shot learners" and nobody bats an eyelid anymore. Which is not to say we have to shut up and take it. I personally draw inspiration from the tale of the little child who pointed out the King's sartorial negligence. ________________ [1] That, in itself, is a title designed to claim some groundbreaking progress not just in technological capabilities but also in scientific understanding. LLMs are few-shot learners, man! Few-Shot!
- mgraczyk 4y ago
  I don't see the issue here. The exact same definition of "few shot learning" has been used for at least 20 years. Nothing changed with the GPT-3 paper. The definition is something like Given a task, a few shot learner is an algorithm that generalizes well with only a small number of training examples for that task. The same definition is what I'm familiar with from undergrad. Do you know of a different definition that precedes GPT-3?
  - mjburgess 4y ago
    The issue is that the aim is to model the one shot of animals, not some other target . Animals are one shot learners because we're in causally direct sensory-motor contact with reality, such that we can dibiguatw it live. No train/predict system can disambiguate the causal origin of data and so can never be one shot. What they're targeting is a triviality within a system of trivialities and misdiscrubjng it
    - mgraczyk 4y ago
      But then you're saying that the definition has always been wrong, which is a very different claim. I personally think that claiming a definition has always been wrong is vacuous. Just substitute the word in your head for something else if you don't like it.
maria2 4y ago
Why should we let philosophy define technical terms for ML? Many words have many meanings. Welcome to the imprecision of human language. As a slight tangent, I really hate this type of comment that inevitably appears on many HN submissions. Personally, I find it distracting when the main conversation happening on an article is a pedantic discussion on whether or not a word means what the author thinks it means.
HarHarVeryFunny 4y ago
I'm OK with the term emergent used here - it seems the best word to describe non-trivial capabiities/properties that weren't designed in. At least prior to the first LLMs, I think most people would just expect them to be capable of doing literally what they are trained to do - predict (locally plausible) next word(s), and this is certainly all that is "designed in" if we're talking about a basic LLM (vs one with further RL-based "alignment", etc). Of course we can also appreciate that to get REALLY REALLY good at "predict next word" would require intelligence/understanding of what is being generated, but I think the point here is would anyone - the model designers in particular - have expected that the transformer architecture (+ scale) is all it would take to become so good at this task? I don't think "attention is all you need" was really anticipating transformers reading API docs and generating code to perform requested functions! One might have expected it to take a much more elaborate and evolved architecture to achieve this level of capability! So, to me, it seems entirely appropriate to describe these models as having emergent capabilities - things they are capable of doing that are really so far above and beyond "predict next word", that it seems churlish and inaccurate to describe them as designed it (or even just confidently predicted).
rafaelero 4y ago
What a waste of time to be worried about how people are using a word.
ot 4y ago
I believe the term is reasonably appropriate here. The abilities being described here are "emergent" in the sense that the model was not specifically trained for them, but they show up anyway. Your example is about modeling a specific function and having its accuracy increase with model complexity, which is classical ML formulation, but this is not what is happening here. LLMs are trained on a very simple task: given a natural text, predict the next word. But as model complexity and training set sizes increase, they start exhibiting more sophisticated abilities, such as basic forms of reasoning, and contextual memory. In your definition, the parts of the whole are "lots of statistics about text" and the emergent property is "semantic reasoning". Scale is inevitably a part of this: somewhere else in the thread you mention that "liquidity" is an emergent property of H2O, but if you take a handful of H2O molecules they don't behave as a liquid.
smeagull 4y ago
People speak in metaphors. I don't know why people get so hung up when it happens. Language isn't a formula unless you're a model. Even then, LLM do cope with some idioms.
xg15 4y ago
> With enough examples, the gap closes... indeed, you can model the addition function (add(x, y) = x + y) just by an infinite sample of its domains. Infinity is not a number. Trainsets cannot be infinite and neither were the number of examples of addition that we humans saw before we became able to do addition on unseen samples. So the question of how finitely many samples you need to generalize a math operation is a different one.

xpe 4y ago

Well said. How did the authors and reviewers miss this?

mjburgess 4y ago
The transition from useless to useful ML models no doubt often seems magical to researchers. But it follows just from the distribution of the training data and from the degree of its compression by the function approximation algorithm they're using. What's "magical" is not their system, but rather that the vast library of text they use for training has useful properties which can be approximated. What researchers are observing is more like the illusion of a "phase transition" in the quality of approximations. This illusion arises because we have discontinuous standards for the approximations. Ie., when assessing free text prediction by LLMs there's very very many ways for them to generate an acceptable answer. For mathematics, there's only one acceptable way. If we applied the same standard/goal to both, no such apparent "quality transition" would occur. LLMs would be exposed as equally good, or equally bad, at prediction regardless of scale.
- xpe 4y ago
  Interesting arguments. It seems plausible and insightful. IMO, your analysis here deserves a longer write-up. Is it something you are working on?
  - mjburgess 4y ago
    Any person with a "scientific attitude" in this field would find it incredibly easy to observer that the training target for natural language is, f(Q) = {A1..An} -- n being very very large and the target for mathematics is, g(Q) = A1 And the model they're using approximates with, m(Q) = A_guess So it's incredibly easy to model f with m because A_guess just has to be close to one of A1...n; and its very hard to model mathematics because it has to be only A1. The reason articles like this are written isn't because people dont know this; it's because they just do not have a sceptical attitude. And that's a problem I can't fix. If they'd approached this issue with the goal of "finding nothing surprising about this behaviour", ie., trying to make it maximally consistent with existing (basic, trivial, widely-taught) theory, they'd reach this conclusion in under 5min. The problem is their goal is always to find something surprising so they can release some PR about it. There's nothing I can write to fix this problem, it's edemic in the whole field. It makes ML/AI research much more like psychology than physics. Alas!
baandang 4y ago
From Introduction To The Theory Of Complex Systems: • Complex systems can exhibit a rich phase structure and have a huge variety of macrostates that often cannot be inferred from the properties of the elements. This is sometimes referred to as emergence. This is the term and as common a term in complex systems as there is. "scale-dependent capabilities" implies inference from elements. I think some people just don't like the very idea even though it is not unlike the concept of stochastic process. I would think the same reasons to not like the concept of emergence applies to stochastic process. Murray Gell-Mann though couldn't even raise complex systems above arguing if emergence is magical thinking so it is probably a lost cause. Such an interesting field that always ends up as this conversation.
- naasking 4y ago
  > "scale-dependent capabilities" implies inference from elements. Don't confuse the post-hoc explanation with an a priori inference. We can post-hoc explain water's emergent liquidity property using modern quantum theories, but that doesn't mean we could have inferred it if given Schrodinger's equation and the atomic structure of hydrogen and oxygen. "Scale-dependent capability" is a post-hoc explanation that of course looks obvious in hindsight, just like liquidity and pressure looks obvious in hindsight once you understand electromagnetism and atomic theory.

PartiallyTyped 4y ago

> that a property of the whole does not obtain just in virtue of properties of its parts taken in isolation Thank you. I have been expressing variants of this for a while. A paper that comes to mind is OpenAI's hide and seek. They claim that cooperation is emergent behaviour, but each agent is playing it's own version of prisoner's dilemma, and thus learn to cooperate.

visarga 4y ago
That model was not learning from language, it was learning from a simulation. When you can use a simulation to produce training data it is possible to have a model discover new abilities all on its own, like AlphaGo.

seydor 4y ago

Eh eh . imagine how neuroscientists feel

mjburgess 4y ago
Well, of late, NS's have been fond of misusing "hallucinate" likewise which means a non-veridical perception. And they're using it to mean a veridical constructed perception. Leading everyone down a path of ever-more mysticism. It would be nice if neuroscientists spoke out against both their own mystical PR and that of AI, but I don't hear it much.
- oidar 4y ago
  I think the word confabulate would be a better match for what ChatGPT does. When people confabulate they are VERY confident about their invented retellings. This matches the attitude that ChatGPT comes across as when it makes up shit.

naasking 4y ago

I think "scale-dependent capability" is a much more precise term for what they're describing, but I'm not sure that that term doesn't fall under the general umbrella of emergent properties. The opening paragraphs of emergent properties in philosophy [1] cites a number of examples that I would argue are comparable to LLM suddenly becoming able to do arithmetic past a certain scale. > In philosophy and esp. the philosophy of science, emergence is a relation between a whole and its parts such that a property of the whole does not obtain just in virtue of properties of its parts taken in isolation. This is not a settled definition. I think everyone can agree that this applies epistemologically, where studying the parts in isolation cannot always yield sufficient information to predict macroscopic properties, but to claim that all properties of the whole do not reduce to the properties of its parts is controversial. For instance, it seems unlikely that we would have predicted H2O's dipole moment and the phenomenon of surface tension just from studying hydrogen atoms and oxygen atoms in isolation, but it would be incorrect to say that surface tension is not the result of the properties of hydrogen and oxygen in isolation. We simply cannot discover all the relevant properties without studying them together. Edit: to clarify, H2O's dipole moment seems obvious in hindsight when we have a good model for what's going on, and analogously, LLM's ability to do arithmetic seems obvious as a scale-dependent property in hindsight, but that doesn't mean it was obvious that this would happen before it was created. [1] https://plato.stanford.edu/entries/properties-emergent/ https://plato.stanford.edu/entries/properties-emergent/

mjburgess 4y ago
Well it's scale dependence is an illusion. It's uniformly able to model Q->SingleAnswer problems, and uniformly able to model Q-> ManyAnswer problems. Basic arithmetic is the former kind, and it becomes useful to people at large scales, ie., accurate on basic arithmetic. This dual condition of "useful-to-people" is the thing introducing the illusion, since it changes depending on what we're modelling. The system isnt acquring any new property. Consider a researcher putting a book on a thin ice-sheet, and then putting a car on it. Here, they're concluding the ice has different properties in each case -- but it doesnt.
- naasking 4y ago

fumeux_fume 4y ago

Is is that big of a deal? The authors explain their definition of emergent abilities at the beginning of the paper.

mjburgess 4y ago
It's a mystification of what's going on -- the term makes it harder to understand, not easier. It's prone to popular misunderstanding, and it seems even to confuse the researchers themselves.

YeGoblynQueenne 4y ago

Thanks for bringing some sense into the debate. It's scandalising to see how the machine learning research community is so ready to jump on to such ... innovative uses of terminology. Take "few shot learning", for instance. OpenAI's preprint introducing GPT-3 was titled "Large Language Models are Few Shot Learners" [1]. This was promptly adopted by the community, even though LLMs need first to be trained on millions of examples before they can accept "few", and they don't even learn from those few (because no weight updates). So it's not really "few shot" and it's not really "learning", and yet, here we are. "Large Language Models are few-shot learners" and nobody bats an eyelid anymore. Which is not to say we have to shut up and take it. I personally draw inspiration from the tale of the little child who pointed out the King's sartorial negligence. ________________ [1] That, in itself, is a title designed to claim some groundbreaking progress not just in technological capabilities but also in scientific understanding. LLMs are few-shot learners, man! Few-Shot!

mgraczyk 4y ago
I don't see the issue here. The exact same definition of "few shot learning" has been used for at least 20 years. Nothing changed with the GPT-3 paper. The definition is something like Given a task, a few shot learner is an algorithm that generalizes well with only a small number of training examples for that task. The same definition is what I'm familiar with from undergrad. Do you know of a different definition that precedes GPT-3?
- mjburgess 4y ago
  The issue is that the aim is to model the one shot of animals, not some other target . Animals are one shot learners because we're in causally direct sensory-motor contact with reality, such that we can dibiguatw it live. No train/predict system can disambiguate the causal origin of data and so can never be one shot. What they're targeting is a triviality within a system of trivialities and misdiscrubjng it
  - mgraczyk 4y ago
    But then you're saying that the definition has always been wrong, which is a very different claim. I personally think that claiming a definition has always been wrong is vacuous. Just substitute the word in your head for something else if you don't like it.

maria2 4y ago

Why should we let philosophy define technical terms for ML? Many words have many meanings. Welcome to the imprecision of human language. As a slight tangent, I really hate this type of comment that inevitably appears on many HN submissions. Personally, I find it distracting when the main conversation happening on an article is a pedantic discussion on whether or not a word means what the author thinks it means.

HarHarVeryFunny 4y ago

I'm OK with the term emergent used here - it seems the best word to describe non-trivial capabiities/properties that weren't designed in. At least prior to the first LLMs, I think most people would just expect them to be capable of doing literally what they are trained to do - predict (locally plausible) next word(s), and this is certainly all that is "designed in" if we're talking about a basic LLM (vs one with further RL-based "alignment", etc). Of course we can also appreciate that to get REALLY REALLY good at "predict next word" would require intelligence/understanding of what is being generated, but I think the point here is would anyone - the model designers in particular - have expected that the transformer architecture (+ scale) is all it would take to become so good at this task? I don't think "attention is all you need" was really anticipating transformers reading API docs and generating code to perform requested functions! One might have expected it to take a much more elaborate and evolved architecture to achieve this level of capability! So, to me, it seems entirely appropriate to describe these models as having emergent capabilities - things they are capable of doing that are really so far above and beyond "predict next word", that it seems churlish and inaccurate to describe them as designed it (or even just confidently predicted).

rafaelero 4y ago

What a waste of time to be worried about how people are using a word.

ot 4y ago

I believe the term is reasonably appropriate here. The abilities being described here are "emergent" in the sense that the model was not specifically trained for them, but they show up anyway. Your example is about modeling a specific function and having its accuracy increase with model complexity, which is classical ML formulation, but this is not what is happening here. LLMs are trained on a very simple task: given a natural text, predict the next word. But as model complexity and training set sizes increase, they start exhibiting more sophisticated abilities, such as basic forms of reasoning, and contextual memory. In your definition, the parts of the whole are "lots of statistics about text" and the emergent property is "semantic reasoning". Scale is inevitably a part of this: somewhere else in the thread you mention that "liquidity" is an emergent property of H2O, but if you take a handful of H2O molecules they don't behave as a liquid.

smeagull 4y ago

People speak in metaphors. I don't know why people get so hung up when it happens. Language isn't a formula unless you're a model. Even then, LLM do cope with some idioms.

xg15 4y ago

> With enough examples, the gap closes... indeed, you can model the addition function (add(x, y) = x + y) just by an infinite sample of its domains. Infinity is not a number. Trainsets cannot be infinite and neither were the number of examples of addition that we humans saw before we became able to do addition on unseen samples. So the question of how finitely many samples you need to generalize a math operation is a different one.