13 ms·

“Emergent” abilities in LLMs actually develop gradually and predictably – study

255 points by Anon84 3y ago

p1esk 3y ago
This is a good paper. Though emergence doesn’t necessarily require a sudden jump in metrics, or unpredictability. New abilities can emerge gradually.
- n2d4 3y ago
  When we talk about "emergence" in machine learning, we talk about those metrics with a sudden jump, as explained in the paper that introduced the term: https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682
  - exe34 3y ago
    Just because somebody overloaded a word doesn't mean we can't correct people who use it wrong. The correct phrase should be "sudden/unexpected emergence". Otherwise the phrase "gradual emergence" would be an oxymoron.
    - n2d4 3y ago
      As another commenter explained, different domains use words differently. The ML paper I linked has been influential enough to use the word "emergent" in machine learning to describe this behaviour, so you're right that, within the context of ML and AI, "gradual emergence" is an oxymoron. Again, I understand this might not align with what you think "emergence" means in the English language, in the same way in which "metal" in physics has a completely different meaning from "metal" in chemistry, "metal" in astrophysics, and "metal" in the English language.
      - exe34 3y ago
        To be fair, I don't think anybody in chemistry or astrophysics is very proud of this kind of overloading. It was done by some Philistines and the rest of us have to live with their defacement of the dictionary.
  - IshKebab 3y ago
    They didn't introduce the term. It's a standard English word: https://en.wiktionary.org/wiki/emergent#English https://en.wiktionary.org/wiki/emergent#English > (philosophy, sciences) Having properties as a whole that are more complex than the properties contributed by each of the components individually. If they've said that it requires a sudden jump then they're trying to redefine the word to mean something it doesn't. But I haven't heard anyone else use that definition. The emergent properties of AI are like "how can it write and entire Python program when we've only ever asked it to predict single words". It doesn't require any sudden jumps.
    - bpshaver 3y ago
      Indeed, I was surprised by the article's assumption that "emergence" is about a sudden jump in ability. I work in AI and ML but my academic background is in the philosophy of mind. When people talk about "emergence" w/r/t AI I assume they're speculating about how consciousness might be an emergent property of a neural network, in the same way some people speculate that our minds are emergent properties of our brains. I think that speculation is foolish, but that is beside the point. It has nothing to do with a sudden jump in abilities, per se.
    - nyrikki 3y ago
      Polysemous words, which sound the same and spell the same but have two or more than two different meanings are common. Each subfield defines rigourous terms and some may be overloaded. It is impossible to have any universal definition of words that are universal, exact definitions have to be domain specific.
      - IshKebab 3y ago
        Profound, but the AI field has not redefined this term, even if one paper tried to.
      - bpshaver 3y ago
        Polysemous words exist of course, but that doesn't mean they're always unproblematic. The best examples, like bank (where the money is) versus bank (sloped piece of land) are unlikely to be confused because there is no domain where they're often used in conjunction. In this context, its charitable to note that some polysemy is at work within the domain (because the application of terms is so new) and so its useful to have these kind of distinction-making conversations. Many confusions arise when people have different senses of a term but insist there is no polysemy and they're all talking about the same thing.
dataking 3y ago
https://archive.is/Z0EOB https://archive.is/Z0EOB
mellosouls 3y ago
OP Should link to the original article which is listed in the first line: https://www.quantamagazine.org/how-quickly-do-large-language-models-learn-unexpected-skills-20240213/ https://www.quantamagazine.org/how-quickly-do-large-language...
- stavros 3y ago
  How does this work? Why is the whole story just copied over?
  - animaomnium 3y ago
    Wired has a deal with Quanta to republish some of their stories. There are other stories from Quanta in Wired; I no longer read Wired, but I believe that this is how I first learned of Quanta.
    - stavros 3y ago
      Ahh, thanks, I didn't realize this was a thing.
  - mellosouls 3y ago
    I don't know but would assume it's marketing-related, the story seen as meeting the demographic requirements of both publications, with an established mutually-beneficial relationship satisfied in the transaction, eg. maybe cheaper content for Wired, bumped readership figures for Quanta.
empath-nirvana 3y ago
That is a terrible headline. It implies that the _abilities_ are a mirage, but it's actually the "emergent" part that might be a mirage -- which is to say it's not an unpredictable "phase transition" but gradual and predictable improvement in ability as the models scale.
- unclenoriega 3y ago
  FWIW, I read the headline as saying the abilities of LLMs are not emergent.
  - anon291 3y ago
    Emergent and breakthrough are not the same quality. Something can be emergent and develop gradually
    - andsoitis 3y ago
      > Something can be emergent and develop gradually Emergent can be defined as the sudden appearance of novel behavior.
      - hdarshane 3y ago
        emergence simply means something the model wasn't trained and expected to produce.
- taneq 3y ago
  And “emergent” in this context usually means “behaviour resulting from the interaction of a large number of individually simple units”, not “suddenly appearing.”
  - hdarshane 3y ago
    This interpretation is most definitely right. As with all things, it's coming down to semantics.
esafak 3y ago
The paper: Are Emergent Abilities of Large Language Models a Mirage? https://arxiv.org/abs/2304.15004 https://arxiv.org/abs/2304.15004
tambourine_man 3y ago
> They find that with a different measuring stick, emergence vanishes. Isn’t that the case for most/all emergent behavior? If you change the scale and watch individual water molecules, you’d see them snapping into a crystal structure one by one instead of a sudden emergent block of ice.
- godelski 3y ago
  Not quite. The problem is that the definition is especially poorly defined in ML. I elaborate more here[0]. You are describing emergence, but not what was claimed when LLMs were said to have emergent abilities (the distinction is explained in the article fwiw) [0] https://news.ycombinator.com/item?id=39812315 https://news.ycombinator.com/item?id=39812315
- n2d4 3y ago
  But knowing the molecule structures at 50C and 75C tells you very little about the freezing point. Different example: If you measure the number of cases of a given virus, it will either spread across the entire globe (R0 > 1, eg. COVID-19) or fail to spread widely (R0 < 1, eg. Ebola). Even though it's not completely binary, it's emergent behaviour because it looks binary. But if you were to measure R0 directly, you'd see a gradual increase, and could predict future variants/vaccine efficiencies/etc much more easily. "Emergent" refers to eg. sigmoids, while "gradual" refers to eg. linear or logarithmic functions.
- throwaway290 3y ago
  Nobody mistakes ice for something sudden emergent though, it's obvious to naked eye that it gradually appears
  - maxcoder4 3y ago
    I don't know what exactly "emergent" means in this context, but doesn't ice appear pretty rapidly? It appears between -1 and 1 degrees, and when cooling down from 100 to 75 to 50 to 25 water you would learn nothing about the freezing point.
wpietri 3y ago
I think the graph is especially interesting. If the progress of LLMs is less radical when measured correctly, that could be another factor driving a bubble and its later popping.
- abound 3y ago
  The updated "partial credit" metric proposed by the researchers suggests that LLMs will continue to improve as parameter count increases (assuming you have the data to make use of XX trillion parameters). It's just showing that it isn't as "step-wise" as other rankings may have indicated.
  - anon291 3y ago
    It still is stepwise for the kinds of use cases being envisioned. For example, it's useful to know your 1 trillion parameter self driving car model is actually 90 percent of the way there, but it's also useful to know that the threshold you need to meet to implement the technology will likely be met at some point when your parameter count increases.
godelski 3y ago
Emergence is a weird topic. If we go with the physicist's common definition [0,1] we'd need to differentiate into weak and strong. As for weak emergence[2] that is de facto. As for strong emergence[3] well... do we have an example of the phenomena anywhere? To be clear, ML people have a different definition which I do not think is entirely useful. Let's look at the wording used in [4] Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence raises the question of whether additional scaling could potentially further expand the range of capabilities of language models Emergence is when quantitative changes in a system result in qualitative changes in behavior. An ability is emergent if it is not present in smaller models but is present in larger models. I'm not sure anything here is actually meaningful. We have very little knowledge about the interpretation of ML models, so it seems to be jumping the gun to say that phenomena cannot be predicted. The aspects are highly coupled with things like training techniques, optimization objectives (or loss functions), optimization methods, not just architectural designs. We've also after more research found smaller capable of performing things that large models can do. So the goal post moves and makes the condition of emergent abilities. [RTFA] So how do we define emergent abilities? Everything static except for the number of parameters? Does this make sense? Changing the number of parameters is an abstracted way of changing the optimization technique or training method. Clearly these are also different by nature of different batch sizes and/or a LR scheduler. So it is hard to be consistent. Not to mention that we are not great at predicting much of what networks can do in general. There's plenty of people working on the subject (though this is a small proportion of total ML researchers, even if we exclude those that are highly application focused). So are we gonna call something emergent if it is just something we don't know about? And who gets to be the one to decide? I've seen abilities called emergent that are clearly a result of training methods like KL Divergence but surprising to individuals who do not have a rigorous statistics and/or metric theory background. It just all seems to be jumping the gun. It's a new field and a new science. I think it is okay if we're willing to admit that our understanding just isn't great yet. There's no need to embellish or over attribute. These models are without a doubt powerful and useful, but at the same time I feel we are happy to greatly exaggerate all aspects about them. And for the life of me, I can't figure out why critiques are interpreted as dismissive. You can call an LLM a stochastic parrot and still think it is a great achievement and useful tool. Moreso, criticism is essential. Criticism gives us direction in research. Hype gives us motivation. But the two have to be in balance. Motivation without direction is an undirected Monte Carlo search (can still work, but we can do better than a drunken man). Overly criticizing (turning into dismissing, such as calling LLMs useless) is just pulling wool over one's eyes. It is the equivalent of lights being turned off and deciding to sit down and give up. When the lights get turned off you should find how to turn them back on. And we have more clues than a pure random process. After all, aren't we trying to make these better? I know I am. It is why I got interested in researching these things in the first place. I just wish we could have moderate hype rather than this ridiculously excessive one (which I think operates in a feedback loop with excessive criticism as it drives polarization). [0] Sabine Hossenfelder (I know HN loves her): https://www.youtube.com/watch?v=bJE6-VTdbjw https://www.youtube.com/watch?v=bJE6-VTdbjw [1] Sean Carrol: https://www.youtube.com/watch?v=0_PdLja-eGQ https://www.youtube.com/watch?v=0_PdLja-eGQ Yes, Sabine and Sean are both controversial characters but they are well known. [2] Weak emergence is where some larger phenomena forms out of smaller phenomena and results in something that the smaller thing can't do. The result requires the interactions of parts of a system. Temperature is a common example because it is generally easier to discuss the aggregated value of all the particles' "jigglyness" rather than each individual particle. The resultant property can be derived from the individual components. Conway's game of life is another common example. But in game theory we'd call a coalition an emergent property since utility is higher in the collective than the sum of each individual. Clearly this is true for ANNs since they are composed of neurons and a single neuron cannot perform these tasks. [3] Strong convergence is about behavior that CANNOT be derived from the individual constituents. [4] https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682
sigsergv 3y ago
It's really interesting question how can we measure emergent abilities like arithmetic operations. We cannot test every operation on every possible combinations of numbers. Instead we must make sure somehow that LLM performing arithmetic operations using corresponding rules and axioms.
- eropple 3y ago
  > Instead we must make sure somehow that LLM performing arithmetic operations using corresponding rules and axioms. It isn't. It's stringing together likely tokens that approximate (often very effectively!) what its data corpus has done in the past. And, relatedly, the best way I've found GPT4 to solve a word problem is to tell it to write some Python code to spit out an answer; the actual computation part is an easier thing to figure out when it's just running code.
  - scarface_74 3y ago
    A very simple example “list the Presidents in the order they were born”. It gets the order wrong unless you tell it to “use Python” https://chat.openai.com/share/4a673ea0-67d3-4256-b57d-dc1cf873d605 https://chat.openai.com/share/4a673ea0-67d3-4256-b57d-dc1cf8...
    - exe34 3y ago
      Does python involve calling "get_us_presidents()"?
      - maxcoder4 3y ago
        You can check the code it generated in the long OP provided (this button is not very visible so I understand if you missed it).
      - scarface_74 3y ago
        I couldn’t see how to get the code to show in the shared link myself. But I did look at the code during the session when I was creating the link. It’s just what you would expect - a dictionary of US Presidents and the year they were born and one line built in Python function to sort the list.
knallfrosch 3y ago
Too bad the article only mentions addition – which shows gradual improvement. Would have loved to see the claim of emergence supported by an example as well. And more importantly, if the measure is boolean, completely wrong or completely right, then, well, I do expect to see a "sudden" jump in ability.
- n2d4 3y ago
  The original emergence paper has multiple "emergent" metrics. [1] Though, the paper this article is about [2] mentions that among the "emergent" tasks they tested, 92% were measuring accuracy on either exact-string-match or multiple choice, so probably just bad metrics. [1] https://arxiv.org/pdf/2206.07682.pdf https://arxiv.org/pdf/2206.07682.pdf [2] https://arxiv.org/pdf/2304.15004.pdf https://arxiv.org/pdf/2304.15004.pdf
ambicapter 3y ago
This is very insightful. Another one of those "obvious in retrospect" ideas. We tend to think of addition as only correct or not but has anyone ever claimed that LLMs have a "discrete" output (might be using the wrong terminology)? It would then make sense that you need to measure performance in a continuous, not discrete way. Otherwise you'll end up with a sort of "aliasing"-type error.
- woopsn 3y ago
  Not quite in retrospect -- the earlier paper on emergence addresses this > It is also important to consider the evaluation metrics used to measure emergent abilities. For instance, using exact string match as the evaluation metric for long-sequence targets may disguise compounding incremental improvements as emergence. Similar logic may apply for multi-step or arithmetic reasoning problems, where models are only scored on whether they get the final answer to a multi-step problem correct, without any credit given to partially correct solutions. - Page 7, https://arxiv.org/pdf/2206.07682.pdf https://arxiv.org/pdf/2206.07682.pdf They say that cross entropy loss in these cases goes down with model size incrementally, well before "emergent" capabilities appear. If so the model is improving (in a sense) even though these capabilities aren't observable below some critical size.
renewiltord 3y ago
Okay, neat. Using a binary metric means that you observe sudden transitions from success to failure. Using a more granular metric means you observe smoother improvement. Logical and meaningful. It does make sense to evaluate things like 3-digit-sum by “how close textually?” And “how close numerically?” and the phase change is an artifact of the actual question being “is fully correct?”
inopinatus 3y ago
Even a stopped clock gives the right time, twice a day. LLMs are plausibility engines. The fundamental hypothesis on trial here is that increasing plausibility corresponds to increasing correctness. This is easily rejected for the human-sourced content that trains them, implying an upper bound on any dependent phenomenon. It follows that simply scaling LLMs will not produce AGI.
- FeepingCreature 3y ago
  You're assuming that humans reliably pursue correctness rather than plausibility. The whole scientific system specifically being designed to suppress plausible-sounding-but-incorrect claims says otherwise.
  - happytiger 3y ago
    That’s an astute observation.
  - Jensson 3y ago
    > The whole scientific system specifically being designed to suppress plausible-sounding-but-incorrect claims says otherwise. But that we have came up with a scientific system is proof that humans actually can tell the difference, we might be bad at it but we still manage to do it. Humans that can't tell the difference are very unreliable and usually shunned in engineering and other STEM fields.
    - layer8 3y ago
      Doesn’t a claim become instantly implausible when we see how it is incorrect? It's only plausible as long as it might still turn out to be correct. Furthermore, sometimes mistakes are made when assessing correctness, even in mathematical proofs. A correctness judgement thus still has a degree of plausibility associated with it. I'm thus not sure if the distinction being made here isn't just one of degree.
      - Jensson 3y ago
anon291 3y ago
The partial credit approach is fine, but if we want to train a model to get the right answer, then it does matter. What I've noticed is that my loss curves on small models for arithmetic will reach a steady state where they're still getting the answer wrong even if some digits are right. Yes you can keep training but the number of training epochs necessary seems to be inversely correlated exponentially with the model size So a model with x parameters is going to take n^2 time as long as a model with 2x parameters At a certain parameter count this basically means it is nigh impossible to get the right answer via training using gradient descent The more parameters, the easier it is to drive to convergence, which is a real, important metric At some point the estimated time for the ability to emerge spontaneously becomes greater than a human lifetime of even the lifetime of all humanity. So in the sense that increasing model size makes it tractable... I think it's fair to say that the ability emerges suddenly enough.
- naasking 3y ago
  It seems like model learning is too optimized for continuity, eg. continuous variables can be subdivided infinitely, where logic and algorithms are firm rather than fuzzy like this. Learning agents arguably need to be able to generalize from fuzzy to firm concepts to properly learn logic and algorithms. Whether that can happen automatically by scaling or whether that requires a fundamental shift is unclear.
  - TeMPOraL 3y ago
    Except if we take humans as example, we do not "generalize from fuzzy to firm". We don't handle firm, we approximate it with fuzzy. Which is why formal logic makes for a fine family of puzzle games, but fares extremely badly at describing reality and reasoning about reality. Better suited mathematical frameworks are probabilistic in nature, and so happen to be fundamentally continuous.
    - naasking 3y ago
      > We don't handle firm We absolutely do handle firm. Mathematics, logic, morality are all examples that have plenty of firm classifications, as is typical, everyday reasoning, eg. I would state that I am naasking in this thread and I am not TeMPOraL, and I would not state that I am 99.999% probably naasking but 0.001% possibly TeMPOraL. > Which is why formal logic makes for a fine family of puzzle games, but fares extremely badly at describing reality and reasoning about reality. The device you're reading this on would suggest quite strongly that this conclusion is wrong.
skeledrew 3y ago
There seems to be some sort of flaw or misalignment in the hypothesis. By my reading, it seems this trio is saying LLMs do arithmetic by predicting the digits, and at intermediate stages there are actually partial predictions, and so there isn't a huge jump, which would then be classifiable as emergent. But I'm thinking that in order for an LLM to generally predict arithmetic results in the way they are designed, there would be an infinite memory requirement as it'd have to "store" an infinite set of completions. So the key may be to find sets of operands that GPT4/Claude Opus are unable to solve, even with hints similar to what a teacher may give a student learning the topic. Until such a set is found, I'd say this capability continues to meet the "emergent" definition, as the only other explanation - that I can think of - is that the models "discovered" how to do arithmetic in a general way.
- famouswaffles 3y ago
  >that I can think of - is that the models "discovered" how to do arithmetic in a general way. I mean..yeah. Maybe i've simply misunderstood but It seems like you're presenting this as an outlandish idea but something like this is what Neural Networks regularly end up doing especially in the limit (as loss trends down to zero). https://cprimozic.net/blog/reverse-engineering-a-small-neural-network/ https://cprimozic.net/blog/reverse-engineering-a-small-neura... https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mec... Nobody is going to find any such set because time and time again, neural networks show us complete memorization is actually more difficult than just learning the distribution (even if imperfectly).
ekez 3y ago
The metric the authors use confuses me. Edit distance seems like a strange way to test if the model understands arithmetic ([1], Figure 3). I think `1+3=3` would be equally as correct as `1+1=9`? Why not consider how far off the model is `abs(actual-expected)`? I wonder if there is an inflection point with that metric. https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682
- n2d4 3y ago
  It depends on how you do arithmetic. If you're a human and you do column addition, 12345+35791=58136 is just as big of a mistake as 48146 (the actual result is 48136). It's just one mistaken column in both. Binary half-adders work the same way. We don't really know how LLMs do arithmetic. Maybe token edit distance would be interesting, but either way it doesn't really change the claim of the paper. Unrelated: The link is incorrect, the one you're referring to is here: https://arxiv.org/pdf/2304.15004.pdf https://arxiv.org/pdf/2304.15004.pdf
  - HarHarVeryFunny 3y ago
    Yeah, and as an aside I wonder how hard it would be to train an LLM to do addition, multiplication, etc, human style? Presumably it should be possible at least in step-by-step style (as substitute for short-term memory), the same way that we do it. Without using an algorithmic approach, it seems an LLM can only learn a bunch of partially correct heuristics, and attempt to generalize over examples. I've played with this a bit in the past, and came to the conclusion that GPT-3 seems to have learnt to compare the size of numbers (whether accurately or via heuristics), and would get the approximate size of an answer right (depending on the task), even if not the actual value right. I seem to recall it also doing this for tasks like asking for a prime number greater than a particular value.
    - pixl97 3y ago
      I mean, is it efficient to teach them addition human style instead of heuristics of when to call the right function? Imagine you could say 'calc' in your brain, and some separate subcomponent of your brain that is far more power efficient could return an answer almost instantly? You would not focus on understanding addition/subtraction, more you would focus on on when to use addition/subtraction/multiplication or whatever.
m3kw9 3y ago
I think it only feels like it is emergent just like how some people thinks LLMs are conscious. It’s emergent because you don’t really understand what it was capable of? Or is it you have a confirmation bias because you don’t say its an emergent flaw when LLMs get some stuff so crazy wrong
29athrowaway 3y ago
Transformers have in-context meta learning. If you give them examples they can perform new tasks. When you run AutoGPT, the outcome is not always predictable.
Lockal 3y ago
Study: scientists prefer metrics which create an illusion that every time-consuming process is gradual and predictable
- maxcoder4 3y ago
  If a process can be measured by a linearly growing metric I'd say it's predictible. And of course scientists try to understand complex processes and make them more predictible. That distinguishes science from magic. You said that like it's a bad thing
a_wild_dandan 3y ago
There are several issues with the study: 1. Replacing pass/fail accuracy with smoother alternatives (e.g token edit distance) could be a terrible proxy for skill, depending on the task. 2. Even by the authors' metrics, they _still_ find a few potentially emergent abilities. 3. Hindsight is 20-20. Yes, we can revisit the data and fiddle until we find transforms that erase emergence from aptitude plots. The fact is, folk used commonplace test accuracy measurements, and the results were unpredictable and surprising. That's the true notable phenomenon. I think there's value in the paper. Just...don't take its conclusions too far.
- Gisbitus 3y ago
  Just like it's mentioned later in the article: it doesn't really matter if you get an addition mostly right. You either get it right or you don't. I still appreciate their effort though, because even after altering the grading system, there were still some emergent abilities.
  - IanCal 3y ago
    > it doesn't really matter if you get an addition mostly right. This is definitely not true in the real world. Approximate solutions are often good enough to answer a question.
    - RandomLensman 3y ago
      But then it isn't "addition" but approximating "addition".
      - Satam 3y ago
        There's always some uncertainty to any kind of answer or computation, it's just at some threshold we unconsciously decide to take it for a fact.
        RandomLensman 3y ago
jonplackett 3y ago
> But with other tasks, the jump in ability wasn’t smooth. The performance remained near zero for a while, then performance jumped. Second paragraph completely contradicts the title given.
- layer8 3y ago
  Continue reading.
hatenberg 3y ago
Phase changes are a natural pattern in complex systems. See state transition in fluids. Depending on how you measure, they look instant or gradual
Valakas_ 3y ago
Small pet-peeve with title: Everything is math/predictable. We just may or may not be able to predict it with current knowledge.
Satam 3y ago
So the models might be getting smarter without discontinuous jumps. And it could just be that we're measuring them in a way that gives no credit for partial answers, thus we end up missing the signs that they've been getting continuously sharper all along. This sounds kind of in line with what I gather Sam Altman's thinking is. They feel they can predict a model's reasoning abilities quite well based just on the size of the training compute and data.
- ActionHank 3y ago
  Sam Altman is the salesman, it's not his thinking, it's the thinking of the many, many experts that work at OpenAI that he parrots.
  - tucnak 3y ago
    Wow, now you're really sticking it up to the man!
    - Jensson 3y ago
      If you are a fan of Sam Altman can't you say what technical problems he solved and in what way he is a technical leader here? Would be much more effective than just an insult.
      - hkt 3y ago
        VC is very often a cult of personality, let's face it. So is most of capitalism when you think about it - Elon Musk, Jeff Bezos, Bill Gates, etc. They're where they are because of vast quantities of unattributed work by other people.
        ActionHank 3y ago
        He still didn’t solve anything. Rocking up and shoving money in your pockets is hardly a skill.
    - 3y ago
vouaobrasil 3y ago
> But with other tasks, the jump in ability wasn’t smooth. The performance remained near zero for a while, then performance jumped. Other studies found similar leaps in ability. Wow, is the title of the submission inaccurate or what!
- redundantly 3y ago
  I believe in this context they mean these abilities are purposely being worked on and brought into existence, they didn't just pop up out of no where. A significant portion of the world was blindsided by the sudden emergence of "AI", but there were people that knew these things were coming.
- gmdrd 3y ago
  That paragraph outlines the findings of previous research, findings that this paper challenges.
hdarshane 3y ago
I think the importance of this study is being downplayed by many. That same spontaneous and "breakthrough" nature of the abilities of LLM that is now nullified, was a major component of many arguments against LLMs and could've possibly stopped us from achieving greater intelligence. I see this as the removal of a possible great filter event of the evolution of LLMs. This is a pretty big concern being nullified.
HarHarVeryFunny 3y ago
It's always easier to predict the future after it's arrived. There may be some emergent features that represent phase changes that are truly difficult, even in retrospect, to predict, but I doubt these are common (does anyone even have a single convincing LLM example?). I think the more normal case is that higher level capabilities are dependent on multiple lower level ones in ways that may be hard to predict. These smooth improvements in the constituent building blocks may often be there, but you'd have to know where to look (i.e. what the critical building blocks are). In order to predict emergent capabilities you'd need to have identified ahead of time what the necessary building blocks are, and maybe done some simulation of what level each of those need would need to be at, to support the anticipated/hoped-for behavior. Of course it's not just about scaling model size and/or data, but also about type and quality of data, where there may be abrupt changes between model versions. It's going to be very difficult ahead of time to analyze what new patterns/manipulations (i.e. building-block capabilities) the model will be able to learn from an updated training set, and therefore what emergent abilities can be built out of those new blocks. I wonder how often the opposite happens, and model designers have been able to successfully identify that "in order to do X, we need capabilities A, B and C, and in order to get capabilities A, B and C, we need new datasets P & Q" ? In a hypothetical case like that they'd have been able to measure progress towards capability X.
- Jensson 3y ago
  > I wonder how often the opposite happens, and model designers have been able to successfully identity that "in order to do X, we need capabilities A, B and C, and in order to get capabilities A, B and C, we need new datasets P & Q" ? In a hypothetical case like that they'd have been able to measure progress towards capability X. As far as I know before transformers that just didn't happen at all since models couldn't contain that many separate skills without them interfering with each other. This whole thing with models having many high quality capabilities is still pretty new. But yeah, I think we need something like that and all the best LLMs today probably already do it, but none have said what they do so that is just speculation.
  - HarHarVeryFunny 3y ago
sjbr 3y ago
So an instance of epistemological emergence instead of ontological emergence.

255 points by Anon84 3y ago

This is a good paper. Though emergence doesn’t necessarily require a sudden jump in metrics, or unpredictability. New abilities can emerge gradually.

n2d4 3y ago
When we talk about "emergence" in machine learning, we talk about those metrics with a sudden jump, as explained in the paper that introduced the term: https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682
- exe34 3y ago
  Just because somebody overloaded a word doesn't mean we can't correct people who use it wrong. The correct phrase should be "sudden/unexpected emergence". Otherwise the phrase "gradual emergence" would be an oxymoron.
  - n2d4 3y ago
    As another commenter explained, different domains use words differently. The ML paper I linked has been influential enough to use the word "emergent" in machine learning to describe this behaviour, so you're right that, within the context of ML and AI, "gradual emergence" is an oxymoron. Again, I understand this might not align with what you think "emergence" means in the English language, in the same way in which "metal" in physics has a completely different meaning from "metal" in chemistry, "metal" in astrophysics, and "metal" in the English language.
    - exe34 3y ago
      To be fair, I don't think anybody in chemistry or astrophysics is very proud of this kind of overloading. It was done by some Philistines and the rest of us have to live with their defacement of the dictionary.
- IshKebab 3y ago
  They didn't introduce the term. It's a standard English word: https://en.wiktionary.org/wiki/emergent#English https://en.wiktionary.org/wiki/emergent#English > (philosophy, sciences) Having properties as a whole that are more complex than the properties contributed by each of the components individually. If they've said that it requires a sudden jump then they're trying to redefine the word to mean something it doesn't. But I haven't heard anyone else use that definition. The emergent properties of AI are like "how can it write and entire Python program when we've only ever asked it to predict single words". It doesn't require any sudden jumps.
  - bpshaver 3y ago
    Indeed, I was surprised by the article's assumption that "emergence" is about a sudden jump in ability. I work in AI and ML but my academic background is in the philosophy of mind. When people talk about "emergence" w/r/t AI I assume they're speculating about how consciousness might be an emergent property of a neural network, in the same way some people speculate that our minds are emergent properties of our brains. I think that speculation is foolish, but that is beside the point. It has nothing to do with a sudden jump in abilities, per se.
  - nyrikki 3y ago
    Polysemous words, which sound the same and spell the same but have two or more than two different meanings are common. Each subfield defines rigourous terms and some may be overloaded. It is impossible to have any universal definition of words that are universal, exact definitions have to be domain specific.
    - IshKebab 3y ago
      Profound, but the AI field has not redefined this term, even if one paper tried to.
    - bpshaver 3y ago
      Polysemous words exist of course, but that doesn't mean they're always unproblematic. The best examples, like bank (where the money is) versus bank (sloped piece of land) are unlikely to be confused because there is no domain where they're often used in conjunction. In this context, its charitable to note that some polysemy is at work within the domain (because the application of terms is so new) and so its useful to have these kind of distinction-making conversations. Many confusions arise when people have different senses of a term but insist there is no polysemy and they're all talking about the same thing.

dataking 3y ago

https://archive.is/Z0EOB https://archive.is/Z0EOB

mellosouls 3y ago

OP Should link to the original article which is listed in the first line: https://www.quantamagazine.org/how-quickly-do-large-language-models-learn-unexpected-skills-20240213/ https://www.quantamagazine.org/how-quickly-do-large-language...

stavros 3y ago
How does this work? Why is the whole story just copied over?
- animaomnium 3y ago
  Wired has a deal with Quanta to republish some of their stories. There are other stories from Quanta in Wired; I no longer read Wired, but I believe that this is how I first learned of Quanta.
  - stavros 3y ago
    Ahh, thanks, I didn't realize this was a thing.
- mellosouls 3y ago
  I don't know but would assume it's marketing-related, the story seen as meeting the demographic requirements of both publications, with an established mutually-beneficial relationship satisfied in the transaction, eg. maybe cheaper content for Wired, bumped readership figures for Quanta.

empath-nirvana 3y ago

That is a terrible headline. It implies that the _abilities_ are a mirage, but it's actually the "emergent" part that might be a mirage -- which is to say it's not an unpredictable "phase transition" but gradual and predictable improvement in ability as the models scale.

unclenoriega 3y ago
FWIW, I read the headline as saying the abilities of LLMs are not emergent.
- anon291 3y ago
  Emergent and breakthrough are not the same quality. Something can be emergent and develop gradually
  - andsoitis 3y ago
    > Something can be emergent and develop gradually Emergent can be defined as the sudden appearance of novel behavior.
    - hdarshane 3y ago
      emergence simply means something the model wasn't trained and expected to produce.
taneq 3y ago
And “emergent” in this context usually means “behaviour resulting from the interaction of a large number of individually simple units”, not “suddenly appearing.”
- hdarshane 3y ago
  This interpretation is most definitely right. As with all things, it's coming down to semantics.

esafak 3y ago

The paper: Are Emergent Abilities of Large Language Models a Mirage? https://arxiv.org/abs/2304.15004 https://arxiv.org/abs/2304.15004

tambourine_man 3y ago

> They find that with a different measuring stick, emergence vanishes. Isn’t that the case for most/all emergent behavior? If you change the scale and watch individual water molecules, you’d see them snapping into a crystal structure one by one instead of a sudden emergent block of ice.

godelski 3y ago
Not quite. The problem is that the definition is especially poorly defined in ML. I elaborate more here[0]. You are describing emergence, but not what was claimed when LLMs were said to have emergent abilities (the distinction is explained in the article fwiw) [0] https://news.ycombinator.com/item?id=39812315 https://news.ycombinator.com/item?id=39812315
n2d4 3y ago
But knowing the molecule structures at 50C and 75C tells you very little about the freezing point. Different example: If you measure the number of cases of a given virus, it will either spread across the entire globe (R0 > 1, eg. COVID-19) or fail to spread widely (R0 < 1, eg. Ebola). Even though it's not completely binary, it's emergent behaviour because it looks binary. But if you were to measure R0 directly, you'd see a gradual increase, and could predict future variants/vaccine efficiencies/etc much more easily. "Emergent" refers to eg. sigmoids, while "gradual" refers to eg. linear or logarithmic functions.
throwaway290 3y ago
Nobody mistakes ice for something sudden emergent though, it's obvious to naked eye that it gradually appears
- maxcoder4 3y ago
  I don't know what exactly "emergent" means in this context, but doesn't ice appear pretty rapidly? It appears between -1 and 1 degrees, and when cooling down from 100 to 75 to 50 to 25 water you would learn nothing about the freezing point.

wpietri 3y ago

I think the graph is especially interesting. If the progress of LLMs is less radical when measured correctly, that could be another factor driving a bubble and its later popping.

abound 3y ago
The updated "partial credit" metric proposed by the researchers suggests that LLMs will continue to improve as parameter count increases (assuming you have the data to make use of XX trillion parameters). It's just showing that it isn't as "step-wise" as other rankings may have indicated.
- anon291 3y ago
  It still is stepwise for the kinds of use cases being envisioned. For example, it's useful to know your 1 trillion parameter self driving car model is actually 90 percent of the way there, but it's also useful to know that the threshold you need to meet to implement the technology will likely be met at some point when your parameter count increases.

godelski 3y ago

Emergence is a weird topic. If we go with the physicist's common definition [0,1] we'd need to differentiate into weak and strong. As for weak emergence[2] that is de facto. As for strong emergence[3] well... do we have an example of the phenomena anywhere? To be clear, ML people have a different definition which I do not think is entirely useful. Let's look at the wording used in [4] Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence raises the question of whether additional scaling could potentially further expand the range of capabilities of language models Emergence is when quantitative changes in a system result in qualitative changes in behavior. An ability is emergent if it is not present in smaller models but is present in larger models. I'm not sure anything here is actually meaningful. We have very little knowledge about the interpretation of ML models, so it seems to be jumping the gun to say that phenomena cannot be predicted. The aspects are highly coupled with things like training techniques, optimization objectives (or loss functions), optimization methods, not just architectural designs. We've also after more research found smaller capable of performing things that large models can do. So the goal post moves and makes the condition of emergent abilities. [RTFA] So how do we define emergent abilities? Everything static except for the number of parameters? Does this make sense? Changing the number of parameters is an abstracted way of changing the optimization technique or training method. Clearly these are also different by nature of different batch sizes and/or a LR scheduler. So it is hard to be consistent. Not to mention that we are not great at predicting much of what networks can do in general. There's plenty of people working on the subject (though this is a small proportion of total ML researchers, even if we exclude those that are highly application focused). So are we gonna call something emergent if it is just something we don't know about? And who gets to be the one to decide? I've seen abilities called emergent that are clearly a result of training methods like KL Divergence but surprising to individuals who do not have a rigorous statistics and/or metric theory background. It just all seems to be jumping the gun. It's a new field and a new science. I think it is okay if we're willing to admit that our understanding just isn't great yet. There's no need to embellish or over attribute. These models are without a doubt powerful and useful, but at the same time I feel we are happy to greatly exaggerate all aspects about them. And for the life of me, I can't figure out why critiques are interpreted as dismissive. You can call an LLM a stochastic parrot and still think it is a great achievement and useful tool. Moreso, criticism is essential. Criticism gives us direction in research. Hype gives us motivation. But the two have to be in balance. Motivation without direction is an undirected Monte Carlo search (can still work, but we can do better than a drunken man). Overly criticizing (turning into dismissing, such as calling LLMs useless) is just pulling wool over one's eyes. It is the equivalent of lights being turned off and deciding to sit down and give up. When the lights get turned off you should find how to turn them back on. And we have more clues than a pure random process. After all, aren't we trying to make these better? I know I am. It is why I got interested in researching these things in the first place. I just wish we could have moderate hype rather than this ridiculously excessive one (which I think operates in a feedback loop with excessive criticism as it drives polarization). [0] Sabine Hossenfelder (I know HN loves her): https://www.youtube.com/watch?v=bJE6-VTdbjw https://www.youtube.com/watch?v=bJE6-VTdbjw [1] Sean Carrol: https://www.youtube.com/watch?v=0_PdLja-eGQ https://www.youtube.com/watch?v=0_PdLja-eGQ Yes, Sabine and Sean are both controversial characters but they are well known. [2] Weak emergence is where some larger phenomena forms out of smaller phenomena and results in something that the smaller thing can't do. The result requires the interactions of parts of a system. Temperature is a common example because it is generally easier to discuss the aggregated value of all the particles' "jigglyness" rather than each individual particle. The resultant property can be derived from the individual components. Conway's game of life is another common example. But in game theory we'd call a coalition an emergent property since utility is higher in the collective than the sum of each individual. Clearly this is true for ANNs since they are composed of neurons and a single neuron cannot perform these tasks. [3] Strong convergence is about behavior that CANNOT be derived from the individual constituents. [4] https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682

sigsergv 3y ago

It's really interesting question how can we measure emergent abilities like arithmetic operations. We cannot test every operation on every possible combinations of numbers. Instead we must make sure somehow that LLM performing arithmetic operations using corresponding rules and axioms.

eropple 3y ago
> Instead we must make sure somehow that LLM performing arithmetic operations using corresponding rules and axioms. It isn't. It's stringing together likely tokens that approximate (often very effectively!) what its data corpus has done in the past. And, relatedly, the best way I've found GPT4 to solve a word problem is to tell it to write some Python code to spit out an answer; the actual computation part is an easier thing to figure out when it's just running code.
- scarface_74 3y ago
  A very simple example “list the Presidents in the order they were born”. It gets the order wrong unless you tell it to “use Python” https://chat.openai.com/share/4a673ea0-67d3-4256-b57d-dc1cf873d605 https://chat.openai.com/share/4a673ea0-67d3-4256-b57d-dc1cf8...
  - exe34 3y ago
    Does python involve calling "get_us_presidents()"?
    - maxcoder4 3y ago
      You can check the code it generated in the long OP provided (this button is not very visible so I understand if you missed it).
    - scarface_74 3y ago
      I couldn’t see how to get the code to show in the shared link myself. But I did look at the code during the session when I was creating the link. It’s just what you would expect - a dictionary of US Presidents and the year they were born and one line built in Python function to sort the list.

knallfrosch 3y ago

Too bad the article only mentions addition – which shows gradual improvement. Would have loved to see the claim of emergence supported by an example as well. And more importantly, if the measure is boolean, completely wrong or completely right, then, well, I do expect to see a "sudden" jump in ability.

n2d4 3y ago
The original emergence paper has multiple "emergent" metrics. [1] Though, the paper this article is about [2] mentions that among the "emergent" tasks they tested, 92% were measuring accuracy on either exact-string-match or multiple choice, so probably just bad metrics. [1] https://arxiv.org/pdf/2206.07682.pdf https://arxiv.org/pdf/2206.07682.pdf [2] https://arxiv.org/pdf/2304.15004.pdf https://arxiv.org/pdf/2304.15004.pdf

ambicapter 3y ago

This is very insightful. Another one of those "obvious in retrospect" ideas. We tend to think of addition as only correct or not but has anyone ever claimed that LLMs have a "discrete" output (might be using the wrong terminology)? It would then make sense that you need to measure performance in a continuous, not discrete way. Otherwise you'll end up with a sort of "aliasing"-type error.

woopsn 3y ago
Not quite in retrospect -- the earlier paper on emergence addresses this > It is also important to consider the evaluation metrics used to measure emergent abilities. For instance, using exact string match as the evaluation metric for long-sequence targets may disguise compounding incremental improvements as emergence. Similar logic may apply for multi-step or arithmetic reasoning problems, where models are only scored on whether they get the final answer to a multi-step problem correct, without any credit given to partially correct solutions. - Page 7, https://arxiv.org/pdf/2206.07682.pdf https://arxiv.org/pdf/2206.07682.pdf They say that cross entropy loss in these cases goes down with model size incrementally, well before "emergent" capabilities appear. If so the model is improving (in a sense) even though these capabilities aren't observable below some critical size.

renewiltord 3y ago

Okay, neat. Using a binary metric means that you observe sudden transitions from success to failure. Using a more granular metric means you observe smoother improvement. Logical and meaningful. It does make sense to evaluate things like 3-digit-sum by “how close textually?” And “how close numerically?” and the phase change is an artifact of the actual question being “is fully correct?”

inopinatus 3y ago

Even a stopped clock gives the right time, twice a day. LLMs are plausibility engines. The fundamental hypothesis on trial here is that increasing plausibility corresponds to increasing correctness. This is easily rejected for the human-sourced content that trains them, implying an upper bound on any dependent phenomenon. It follows that simply scaling LLMs will not produce AGI.

FeepingCreature 3y ago
You're assuming that humans reliably pursue correctness rather than plausibility. The whole scientific system specifically being designed to suppress plausible-sounding-but-incorrect claims says otherwise.
- happytiger 3y ago
  That’s an astute observation.
- Jensson 3y ago
  > The whole scientific system specifically being designed to suppress plausible-sounding-but-incorrect claims says otherwise. But that we have came up with a scientific system is proof that humans actually can tell the difference, we might be bad at it but we still manage to do it. Humans that can't tell the difference are very unreliable and usually shunned in engineering and other STEM fields.
  - layer8 3y ago
    Doesn’t a claim become instantly implausible when we see how it is incorrect? It's only plausible as long as it might still turn out to be correct. Furthermore, sometimes mistakes are made when assessing correctness, even in mathematical proofs. A correctness judgement thus still has a degree of plausibility associated with it. I'm thus not sure if the distinction being made here isn't just one of degree.
    - Jensson 3y ago

anon291 3y ago

The partial credit approach is fine, but if we want to train a model to get the right answer, then it does matter. What I've noticed is that my loss curves on small models for arithmetic will reach a steady state where they're still getting the answer wrong even if some digits are right. Yes you can keep training but the number of training epochs necessary seems to be inversely correlated exponentially with the model size So a model with x parameters is going to take n^2 time as long as a model with 2x parameters At a certain parameter count this basically means it is nigh impossible to get the right answer via training using gradient descent The more parameters, the easier it is to drive to convergence, which is a real, important metric At some point the estimated time for the ability to emerge spontaneously becomes greater than a human lifetime of even the lifetime of all humanity. So in the sense that increasing model size makes it tractable... I think it's fair to say that the ability emerges suddenly enough.

naasking 3y ago
It seems like model learning is too optimized for continuity, eg. continuous variables can be subdivided infinitely, where logic and algorithms are firm rather than fuzzy like this. Learning agents arguably need to be able to generalize from fuzzy to firm concepts to properly learn logic and algorithms. Whether that can happen automatically by scaling or whether that requires a fundamental shift is unclear.
- TeMPOraL 3y ago
  Except if we take humans as example, we do not "generalize from fuzzy to firm". We don't handle firm, we approximate it with fuzzy. Which is why formal logic makes for a fine family of puzzle games, but fares extremely badly at describing reality and reasoning about reality. Better suited mathematical frameworks are probabilistic in nature, and so happen to be fundamentally continuous.
  - naasking 3y ago
    > We don't handle firm We absolutely do handle firm. Mathematics, logic, morality are all examples that have plenty of firm classifications, as is typical, everyday reasoning, eg. I would state that I am naasking in this thread and I am not TeMPOraL, and I would not state that I am 99.999% probably naasking but 0.001% possibly TeMPOraL. > Which is why formal logic makes for a fine family of puzzle games, but fares extremely badly at describing reality and reasoning about reality. The device you're reading this on would suggest quite strongly that this conclusion is wrong.

skeledrew 3y ago

There seems to be some sort of flaw or misalignment in the hypothesis. By my reading, it seems this trio is saying LLMs do arithmetic by predicting the digits, and at intermediate stages there are actually partial predictions, and so there isn't a huge jump, which would then be classifiable as emergent. But I'm thinking that in order for an LLM to generally predict arithmetic results in the way they are designed, there would be an infinite memory requirement as it'd have to "store" an infinite set of completions. So the key may be to find sets of operands that GPT4/Claude Opus are unable to solve, even with hints similar to what a teacher may give a student learning the topic. Until such a set is found, I'd say this capability continues to meet the "emergent" definition, as the only other explanation - that I can think of - is that the models "discovered" how to do arithmetic in a general way.

famouswaffles 3y ago
>that I can think of - is that the models "discovered" how to do arithmetic in a general way. I mean..yeah. Maybe i've simply misunderstood but It seems like you're presenting this as an outlandish idea but something like this is what Neural Networks regularly end up doing especially in the limit (as loss trends down to zero). https://cprimozic.net/blog/reverse-engineering-a-small-neural-network/ https://cprimozic.net/blog/reverse-engineering-a-small-neura... https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mec... Nobody is going to find any such set because time and time again, neural networks show us complete memorization is actually more difficult than just learning the distribution (even if imperfectly).

ekez 3y ago

The metric the authors use confuses me. Edit distance seems like a strange way to test if the model understands arithmetic ([1], Figure 3). I think `1+3=3` would be equally as correct as `1+1=9`? Why not consider how far off the model is `abs(actual-expected)`? I wonder if there is an inflection point with that metric. https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682

n2d4 3y ago
It depends on how you do arithmetic. If you're a human and you do column addition, 12345+35791=58136 is just as big of a mistake as 48146 (the actual result is 48136). It's just one mistaken column in both. Binary half-adders work the same way. We don't really know how LLMs do arithmetic. Maybe token edit distance would be interesting, but either way it doesn't really change the claim of the paper. Unrelated: The link is incorrect, the one you're referring to is here: https://arxiv.org/pdf/2304.15004.pdf https://arxiv.org/pdf/2304.15004.pdf
- HarHarVeryFunny 3y ago
  Yeah, and as an aside I wonder how hard it would be to train an LLM to do addition, multiplication, etc, human style? Presumably it should be possible at least in step-by-step style (as substitute for short-term memory), the same way that we do it. Without using an algorithmic approach, it seems an LLM can only learn a bunch of partially correct heuristics, and attempt to generalize over examples. I've played with this a bit in the past, and came to the conclusion that GPT-3 seems to have learnt to compare the size of numbers (whether accurately or via heuristics), and would get the approximate size of an answer right (depending on the task), even if not the actual value right. I seem to recall it also doing this for tasks like asking for a prime number greater than a particular value.
  - pixl97 3y ago
    I mean, is it efficient to teach them addition human style instead of heuristics of when to call the right function? Imagine you could say 'calc' in your brain, and some separate subcomponent of your brain that is far more power efficient could return an answer almost instantly? You would not focus on understanding addition/subtraction, more you would focus on on when to use addition/subtraction/multiplication or whatever.

m3kw9 3y ago

I think it only feels like it is emergent just like how some people thinks LLMs are conscious. It’s emergent because you don’t really understand what it was capable of? Or is it you have a confirmation bias because you don’t say its an emergent flaw when LLMs get some stuff so crazy wrong

29athrowaway 3y ago

Transformers have in-context meta learning. If you give them examples they can perform new tasks. When you run AutoGPT, the outcome is not always predictable.

Lockal 3y ago

Study: scientists prefer metrics which create an illusion that every time-consuming process is gradual and predictable

maxcoder4 3y ago
If a process can be measured by a linearly growing metric I'd say it's predictible. And of course scientists try to understand complex processes and make them more predictible. That distinguishes science from magic. You said that like it's a bad thing

a_wild_dandan 3y ago

There are several issues with the study: 1. Replacing pass/fail accuracy with smoother alternatives (e.g token edit distance) could be a terrible proxy for skill, depending on the task. 2. Even by the authors' metrics, they _still_ find a few potentially emergent abilities. 3. Hindsight is 20-20. Yes, we can revisit the data and fiddle until we find transforms that erase emergence from aptitude plots. The fact is, folk used commonplace test accuracy measurements, and the results were unpredictable and surprising. That's the true notable phenomenon. I think there's value in the paper. Just...don't take its conclusions too far.

Gisbitus 3y ago
Just like it's mentioned later in the article: it doesn't really matter if you get an addition mostly right. You either get it right or you don't. I still appreciate their effort though, because even after altering the grading system, there were still some emergent abilities.
- IanCal 3y ago
  > it doesn't really matter if you get an addition mostly right. This is definitely not true in the real world. Approximate solutions are often good enough to answer a question.
  - RandomLensman 3y ago
    But then it isn't "addition" but approximating "addition".
    - Satam 3y ago
      There's always some uncertainty to any kind of answer or computation, it's just at some threshold we unconsciously decide to take it for a fact.
      - RandomLensman 3y ago

jonplackett 3y ago

> But with other tasks, the jump in ability wasn’t smooth. The performance remained near zero for a while, then performance jumped. Second paragraph completely contradicts the title given.

layer8 3y ago
Continue reading.

hatenberg 3y ago

Phase changes are a natural pattern in complex systems. See state transition in fluids. Depending on how you measure, they look instant or gradual

Valakas_ 3y ago

Small pet-peeve with title: Everything is math/predictable. We just may or may not be able to predict it with current knowledge.

Satam 3y ago

So the models might be getting smarter without discontinuous jumps. And it could just be that we're measuring them in a way that gives no credit for partial answers, thus we end up missing the signs that they've been getting continuously sharper all along. This sounds kind of in line with what I gather Sam Altman's thinking is. They feel they can predict a model's reasoning abilities quite well based just on the size of the training compute and data.

ActionHank 3y ago
Sam Altman is the salesman, it's not his thinking, it's the thinking of the many, many experts that work at OpenAI that he parrots.
- tucnak 3y ago
  Wow, now you're really sticking it up to the man!
  - Jensson 3y ago
    If you are a fan of Sam Altman can't you say what technical problems he solved and in what way he is a technical leader here? Would be much more effective than just an insult.
    - hkt 3y ago
      VC is very often a cult of personality, let's face it. So is most of capitalism when you think about it - Elon Musk, Jeff Bezos, Bill Gates, etc. They're where they are because of vast quantities of unattributed work by other people.
      - ActionHank 3y ago
        He still didn’t solve anything. Rocking up and shoving money in your pockets is hardly a skill.
  - 3y ago

vouaobrasil 3y ago

> But with other tasks, the jump in ability wasn’t smooth. The performance remained near zero for a while, then performance jumped. Other studies found similar leaps in ability. Wow, is the title of the submission inaccurate or what!

redundantly 3y ago
I believe in this context they mean these abilities are purposely being worked on and brought into existence, they didn't just pop up out of no where. A significant portion of the world was blindsided by the sudden emergence of "AI", but there were people that knew these things were coming.
gmdrd 3y ago
That paragraph outlines the findings of previous research, findings that this paper challenges.

hdarshane 3y ago

I think the importance of this study is being downplayed by many. That same spontaneous and "breakthrough" nature of the abilities of LLM that is now nullified, was a major component of many arguments against LLMs and could've possibly stopped us from achieving greater intelligence. I see this as the removal of a possible great filter event of the evolution of LLMs. This is a pretty big concern being nullified.

HarHarVeryFunny 3y ago

It's always easier to predict the future after it's arrived. There may be some emergent features that represent phase changes that are truly difficult, even in retrospect, to predict, but I doubt these are common (does anyone even have a single convincing LLM example?). I think the more normal case is that higher level capabilities are dependent on multiple lower level ones in ways that may be hard to predict. These smooth improvements in the constituent building blocks may often be there, but you'd have to know where to look (i.e. what the critical building blocks are). In order to predict emergent capabilities you'd need to have identified ahead of time what the necessary building blocks are, and maybe done some simulation of what level each of those need would need to be at, to support the anticipated/hoped-for behavior. Of course it's not just about scaling model size and/or data, but also about type and quality of data, where there may be abrupt changes between model versions. It's going to be very difficult ahead of time to analyze what new patterns/manipulations (i.e. building-block capabilities) the model will be able to learn from an updated training set, and therefore what emergent abilities can be built out of those new blocks. I wonder how often the opposite happens, and model designers have been able to successfully identify that "in order to do X, we need capabilities A, B and C, and in order to get capabilities A, B and C, we need new datasets P & Q" ? In a hypothetical case like that they'd have been able to measure progress towards capability X.

Jensson 3y ago
> I wonder how often the opposite happens, and model designers have been able to successfully identity that "in order to do X, we need capabilities A, B and C, and in order to get capabilities A, B and C, we need new datasets P & Q" ? In a hypothetical case like that they'd have been able to measure progress towards capability X. As far as I know before transformers that just didn't happen at all since models couldn't contain that many separate skills without them interfering with each other. This whole thing with models having many high quality capabilities is still pretty new. But yeah, I think we need something like that and all the best LLMs today probably already do it, but none have said what they do so that is just speculation.
- HarHarVeryFunny 3y ago

sjbr 3y ago

So an instance of epistemological emergence instead of ontological emergence.