11 ms·
Why do LLMs have emergent properties?
- cratermoon 1y agoAlternate view: Are Emergent Abilities of Large Language Models a Mirage? https://arxiv.org/abs/2304.15004 https://arxiv.org/abs/2304.15004 "Here, we present an alternative explanation for emergent abilities: that for a particular task and model family, when analyzing fixed model outputs, emergent abilities appear due to the researcher's choice of metric rather than due to fundamental changes in model behavior with scale. Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous predictable changes in model performance."
- moffkalast 1y agoThat has been a problem with most LLM benchmarks. Any test that's rated in percentages tends to be logarithmic, since getting from say 90% to 95% is not a linear 5% improvement but probably more like a 2x or 10x improvement in practical terms, since the metric is already nearly maxed out and only the extreme edge cases remain that are much harder to master.
- K0balt 1y agoA decent thought-proxy for this : powered flight. An aircraft can approach powered flight without achieving it. With a given amount of thrust or aerodynamic characteristics, the aircraft will weigh dynamic_weight=(static_weight - x) where x is a combination of the aerodynamic characteristics and the amount of thrust applied. In no case where dynamic_weight>0 will the aircraft fly, even though it exhibits characteristics of flight, I.e the transfer of aerodynamic forces to counteract gravity. So while it progressively exhibits characteristics of flight, it is not capable of any kind of flight at all until the critical point of dynamic_weight<0. So the enabling characteristics are not “emergent”, but the behavior is. I think this boils down to a matter of semantics.
- scopemouthwash 1y ago“Thought-proxy”? I think the word you’re looking for is “analogy”.
- K0balt 1y agoAnalogy is a great word proxy for thought proxy. Yeah, I’m pretty sure analogy would have been fine there, I think maybe it fell off the edge of my vocabulary for a moment? Not really sure, but I really can’t think of any reason why “thought proxy” would have been more descriptive, informative, or accurate ¯_(ツ)_/¯
- foobarqux 1y agoThe author himself explicitly acknowledges the paper but the incomprehensibly ignores it ("Even so, many would like to understand, predict, and even facilitate the emergence of these capabilities."). It's like saying "some say [foo] doesn't exist but even so many would like to understand [foo]". It's incoherent.
- autoexec 1y agoNo point in letting facts get in the way of an entire article I guess.
- jebarker 1y agoYes, this paper is under-appreciated. The point is that we as humans decide what constitutes a given task we're going to set as a bar and it turns out that statistical pattern matching can solve many of those tasks to a reasonable level (we also get to define "reasonable") when there's sufficient scale of parameters and data, but that tip-over point is entirely arbitrary.
- Al-Khwarizmi 1y agoThe continuous metrics the paper uses are largely irrelevant in practice, though. The sudden changes appear when you use metrics people actually care about. To me the paper is overhyped. Knowing how neural networks work, it's clear that there are going to be underlying properties that vary smoothly. This doesn't preclude the existence of emergent abilities.
- chasing0entropy 1y agoThere are eerie similarities in radiographs of LLM inference output and mammalian EEGs. I would be surprised not see latent and surprisingly complicated characteristics become apparent as context and recursive algorithms grow larger.
- aeonik 1y agoWhat graphs are you talking about? I've never heard of LLM radiographs, and my searches are coming up empty.
- unsupp0rted 1y agoIsn't "emergent properties" another way to say "we're not very good at understanding the capabilities of complex systems"?
- seliopou 1y agoYes, it’s a cop-out and smells mostly of dualism: https://plato.stanford.edu/entries/properties-emergent/ https://plato.stanford.edu/entries/properties-emergent/
- theobreuerweil 1y agoI understood it to mean properties of large-scale systems that are not properties of its components. Like in thermodynamics: zooming in to a molecular level, you can reverse time without anything seeming off. Suddenly you get a trillion molecules and things like entropy appear, and time is not reversible at all.
- tunesmith 1y agoIt just means they haven't modeled the externalities. A plane on the ground isn't emergent. In the air it is, at least until you perfectly model weather, which you can't do, so its behavior is emergent. But I think a plane is also a good comparison because it shows that you can manage it; we don't have to perfectly model weather to still have fairly predictable air travel.
- esafak 1y agoNot quite. Complex systems can exhibit macroscopic properties not evident at microscopic scales. For example, birds self organize into flocks, an emergent phenomenon, visible to the untrained eye. Our understanding of how it happens does not change the fact that it does. There is a field of study for this called statistical mechanics. https://ganguli-gang.stanford.edu/pdf/20.StatMechDeep.pdf https://ganguli-gang.stanford.edu/pdf/20.StatMechDeep.pdf
- HPsquared 1y agoVery interesting crossover!
- Michelangelo11 1y agoHow could they not? Emergent properties are unavoidable for any complex system and probably exponentially scale with complexity or something (I'm sure there's an entire literature about this somewhere). One good instance are spandrels in evolutionary biology. The wikipedia article is a good explanation of the subject: https://en.m.wikipedia.org/wiki/Spandrel_(biology) https://en.m.wikipedia.org/wiki/Spandrel_(biology)
- deleted 1y ago[deleted]
- creer 1y agoThis is also my impression "how could they not" but it goes a bit further: Can we predict it? Can we estimate the size of a system that will achieve X? Can we build systems that are pre-disposed to emergent behaviors? Can we build systems that preclude some class of emergent behavior (relevant to AGI safety perhaps)? And then of course many systems will not achieve anything because even when "large", they are "uselessly large" - as in, you can define more points on a line and it's still a dumb line. To me the "how could they not" comes from the idea that if LLMs somehow encapsulate/ contain/ exploit all human writings, then they most likely cover a large part of human intelligence. For damn sure much more than the "basic human". The question is more of how we can get this behavior back out of it - than whether it's there.
- EGreg 1y ago[flagged]
- OtherShrezzing 1y agoIt feels like this can be tracked with addition. Humans expect “can do addition” is a binary skill, because humans either can or cannot add. LLMs approximate addition. For a long time they would produce hot garbage. Then after a lot of training, they could sum 2 digit numbers correctly. At this point we’d say “they can do addition”, and the property has emerged. They have passed a binary skill threshold.
- me3meme 1y agoMetaphor: finding a path from a initial point to a destination in a graph. As the number of parameters increases one can expect the LLM to be able to remember how to go from one place to another and in the end it should be able to find a long path. This can be an emergent property since with less parameters the LLM could not be able to find the correct path. Now one has to find what kind of problems this metaphor is a good model of.
- deleted 1y ago[deleted]
- andy99 1y agoI didn't follow entirely on a fast read, but this confused me especially: The parameter count of an LLM defines a certain bit budget. This bit budget must be spread across many, many tasks I'm pretty sure that LLMs, like all big neural networks, are massively under-specified, as in there are way more parameters than data to fit (understanding the training data set is bigger than the size of the model, but the point is the same loss can be achieved with many different combinations of parameters). And I think of this underspecification as the reason neural networks extrapolate cleanly and this generalize.
- vonneumannstan 1y agoThis doesn't seem right and most people recognize that 'neurons' encode for multiple activations. https://transformer-circuits.pub/2022/toy_model/index.html https://transformer-circuits.pub/2022/toy_model/index.html
- Der_Einzige 1y agoThey’re 1000% right on the idea that most models are hilariously undertrained
- vonneumannstan 1y agoPretty sure that since the Chinchilla paper this probably isn't the case. https://arxiv.org/pdf/2203.15556 https://arxiv.org/pdf/2203.15556
- kazinator 1y ago> the same loss can be achieved with many different combinations of parameters Perhaps it can be, but it isn't. The loss in a given model was achieved with that particular combination of parameters that the model has, and there exists no other combination of parameters which which that model can appeal to for more information. To have the other combinations, we would need to train more models and then somehow combine them so that the combinations are available; but that's just conceptually the same as making one larger model with more parameters, and lower loss.
- lordnacho 1y agoWhat seems a bit miraculous to me is, how did the researchers who put us on this path come to suspect that you could just throw more data and more parameters at the problem? If the emergent behavior doesn't appear for moderate sized models, how do you convince management to let you build a huge model?
- educasean 1y agoYou might appreciate this article: https://www.quantamagazine.org/when-chatgpt-broke-an-entire-field-an-oral-history-20250430/ https://www.quantamagazine.org/when-chatgpt-broke-an-entire-...
- TheCoreh 1y agoThis is perhaps why it took us this long to get to LLMs, the underlying math and ideas were (mostly) there, and even if the Transformer as an architecture wasn't ready yet, it wouldn't surprise me if throwing sufficient data/compute at a worse architecture wouldn't also produce comparable emergent behavior There needed to be someone willing to try going big at an organization with sufficient idle compute/data just sitting there, not a surprise it first happened at Google.
- hibikir 1y agoBut we got here step by step, as other interesting use cases came up by using somewhat less compute. Image recognition, early forms of image generation, AlphaGo, AlphaZero for chess. All earlier forms of deep neural networks that are much more reasonable than training a top of the line LLM today, but seemed expensive at the time. And ultimately a lot of this also comes from the hardware advancements and the math advancements. If you took classes neural networks in the 1990s, you'd notice that they mostly talked about 1 or 2 hidden layers, and not all that much focus on the math to train large networks, precisely because of how daunting the compute costs were for anything that wasn't a toy. But then came video card hardware, and improvements to use it to do gradient descent, making going past silly 3 layer networks somewhat reasonable. Every bet makes perfect sense after you consider how promising the previous one looked, and how much cheaper the compute was getting. Imagine being tasked to train an LLM in 1995: All the architectural knowledge we have today and a state-level mandate would not have gotten all that far. Just the amount of fast memory that we put to bear wouldn't have been viable until relatively recently.
- waynecochran 1y agoSince gradient descent converges on a local minima, would we expect different emergent properties with different initialization of the weights?
- jebarker 1y agoNot significantly, as I understand it. There's certainly variation in LLM abilities with different initializations but the volume and content of the data is a far bigger determinant of what an LLM will learn.
- waynecochran 1y agoSo there is an "attractor" that different initializations end up converging on?
- andy99 1y agoDifferent initialization converge to different places, e.g https://arxiv.org/abs/1912.02757 https://arxiv.org/abs/1912.02757 For LLMs (as with other models), many local optima appear to support roughly the same behavior. This is the idea of the problem being under-specified ie many more equations than unknowns so there are many ways to get the same result.
- recursivecaveat 1y agoYou end up with different weights when using different random initialization, but with modern techniques the behavior of the resulting model is not really distinct. Back in the image-recognition days it was like +/- 0.5% accuracy. If you imagine you're descending in a billion-parameter space, you will always have a negative gradient to follow in some dimension: local minima frequency goes down rapidly with (independent) dimension count.
- juancn 1y agoI always wondered if the specific dimensionality of the layers and tensors has a specific effect on the model. It's hard to explain, but higher dimensional spaces have weird topological properties, not all behave the same way and some things are perfectly doable in one set of dimensions while for others it just plain doesn't work (e.g. applying surgery on to turn a shape into another).
- etrautmann 1y agoHow is topology specifically related to emergent capabilities in AI?
- scopemouthwash 1y agoThe authors haven’t demonstrated emergence of LLMs. If I write a piece of code and it does what I programmed it to do that’s not emergence. LLMs aren’t doing anything unexpected yet. I think that’s the smell test because emergence is still subjective.
- teekert 1y agoPerhaps we should ask: Why do humans pick arbitrary points on a continuum beyond which things are labeled “emergent”?
- deleted 1y ago[deleted]
- creer 1y agoSure but also see the benchmark creation to benchmark breaking race. The benchmark creation researchers have been doing their best to create difficult, lasting benchmarks that won't be broken by next week. They are not illiterate idiots. And yet the LLMs (or LLM-based systems really) have been consistently so far breaking the benchmarks in absurd time. It's hard to say that nothing significant is going on. Also the fine article's entire point is that these are not points on a continuum.
- Jensson 1y ago> The benchmark creation researchers have been doing their best to create difficult, lasting benchmarks that won't be broken by next week. That is really easy, say make it play pokemon as well as a 10 year old. That would take a very long time, I watched gemini play pokemon and its nowhere close to that even with all that help. The hard part isn't making a benchmark that wont be broken, its making a benchmark that is so easy that LLM can solve them but that is still hard for them to solve. Essentially what this means is that we have ran out of easy progress, and are now stumbling in the dark since we have no effective benchmarks to chase.
- nthingtohide 1y agoWhat do you think about this analogy? A simple process produces a Mandelbrot set. A simple process (loss minimization through gradient descent) produces LLMs. So what plays the role of 2D-plane or dense point grid in the case of LLMs? It is the embeddings, (or ordered combinations of embeddings ) which are generated after pre-training. In case of a 2D plan, the closeness between two points is determined by our numerical representation schema. But in case of embeddings, we learn the 2D-grid of words (playing the role of points) by looking at how the words are getting used in corpus The following is a quote from Yuri Manin, an eminent Mathematician. https://www.youtube.com/watch?v=BNzZt0QHj9U https://www.youtube.com/watch?v=BNzZt0QHj9U Of the properties of mathematics, as a language, the most peculiar one is that by playing formal games with an input mathematical text, one can get an output text which seemingly carries new knowledge. The basic examples are furnished by scientific or technological calculations: general laws plus initial conditions produce predictions, often only after time-consuming and computer-aided work. One can say that the input contains an implicit knowledge which is thereby made explicit. I have a related idea which I picked up from somewhere which mirrors the above observation. When we see beautiful fractals generated by simple equations and iterative processes, we give importance to only the equations, not to the cartesian grid on which that process operates.
- deleted 1y ago[deleted]
- pixl97 1y ago> the most peculiar one is that by playing formal games with an input mathematical text, one can get an output text which seemingly carries new knowledge. Or biologically, DNA/RNA behaves in a similar manner.
- RigelKentaurus 1y agoI'm not a techie, so perhaps someone can help me understand this: AFAIK, no theoretical computer scientist predicted emergence in AI models. Doesn't that suggest that the field of theoretical computer science (or theoretical AI, if you will) is suspect? It's like Lord Kelvin saying that heavier-than-air flying machines are impossible a decade before the Wright brothers' first flight.
- waynecochran 1y agohttps://hai.stanford.edu/news/ais-ostensible-emergent-abilities-are-mirage https://hai.stanford.edu/news/ais-ostensible-emergent-abilit...
- xboxnolifes 1y agoHas emergent behavior ever been predicted prior to it being observed in other theoretical fields?
- tinix 1y agoemergent behavior is common in all large systems. it doesn't seem that surprising to me.
- xboxnolifes 1y agoThat is not the question.
- tinix 1y agoMaybe I can make this more clear for you: Yes, the answer is yes. lol see: physics, biology, economics, sociology more granularly: cellular automata, phase transitions, trophic cascades, scale-free networks, swarm intelligence, the list goes on... all of this was modeled theoretically before it was observed.
- pixl97 1y agoI believe it's been predicted in traffic planning and highway design and tested in via simulation and in field experiments. Use of self driving cars to modify traffic behaviors and decrease traffic jams is a field of study these days.
- interstice 1y agoThe bag of heuristics thing is interesting to me, is it not conceivable that a NN of a certain size trained only on math problems would be able to wire up what amounts to a calculator? And if so, could that form part of a wider network, or is I/O from completely different modalities not really possible in this way?
- anon373839 1y agoI remain skeptical of emergent properties in LLMs in the way that people have used that term. There was a belief 3-4 years ago that if you just make the models big enough, they magically acquire intelligence. But since then, we’ve seen that the models are actually still pretty limited by the training data: like other ML models, they interpolate well between the data they’ve been trained on, but they don’t generalize well beyond it. Also, we have seen models that are 50-100x smaller now exhibit the same “emergent” capabilities that were once thought to require hundreds of billions of parameters. I personally think the emergent properties really belong to the data instead.
- andy99 1y agoYes, deep learning models only interpolate, and essentially represent an effective way of storing data labeling effort. Doesn't mean they're not useful, just not what tech adjacent promoters want people to think.
- john-h-k 1y ago> Yes, deep learning models only interpolate What do you mean by this? I don’t think the understanding of LLMs is sufficient to make this claim
- andy99 1y agoAn LLM is a classifier, there is lots of research into how deep learning classifiers work, that I haven't seen contradicted when applied to LLMs.
- drdeca 1y agoI still think it seems unclear what you mean by “interpolate” in this context? If your NN takes in several numbers and assigns logits to each class based on those numbers, then if you consider the n dimensional space of possible inputs, and if the new input is in the convex hull of the inputs that appear in training samples, then the meaning of “interpolate” is fairly clear. But when the inputs are sequences of tokens… Granted, each token gets embedded as some vector, and you can concatenate those vectors to represent the sequence of tokens as one big vector, but, are these vectors for novel strings in the convex hull of such vectors for the strings in the training set?
- gyomu 1y agoThe reasoning in the article is interesting, but this struck me as a weird example to choose: > “The real question is how can we predict when a new LLM will achieve some new capability X. For example, X = “Write a short story that resonates with the social mood of the present time and is a runaway hit” Framing a capability as something that is objectively measurable (“able to perform math on the 12th grade level”, “able to write a coherent, novel text without spelling/grammar mistakes”) makes sense within the context of what the author is trying to demonstrate. But the social proof aspect (“is a runaway hit”) feels orthogonal to it? Things can be runaway hits for social factors independently of the capability they actually represent.
- Retric 1y agoIt’s not about being “a runaway hit” as an objective measurement it’s about the things an LLM would need to achieve before that was possible. At first AI scores on existing tests seemed like a useful metric. However, tests designed for humans make specific assumptions that don’t apply to these systems making such tests useless. AI is very good at gaming metrics so it’s difficult to list some criteria where achieving it is meaningful. A hypothetical coherent novel without spelling/grammar mistakes could in effect be a copy of some existing work that shows up in its corpus, however a hit requires more than a reskinned story.
- chii 1y ago> however a hit requires more than a reskinned story. demonstrably false with a lot of hits in the past that is a reskinned story of existing stories!
- Retric 1y agoWhile not a technical term of art, copyright applies to a reskinned story. “the series is not available in English translation, because of the first book having been judged a breach of copyright.” https://en.wikipedia.org/wiki/Tanya_Grotter https://en.wikipedia.org/wiki/Tanya_Grotter There’s plenty of room to take inspiration and go in another direction aka Pride and Prejudice and Zombies.
- samirillian 1y ago*Do
- zmmmmm 1y agoThis seems superficial and doesn't really get to the heart of the question. To me it's not so much about bits and parameters but a more interesting fundamental question of whether pure language itself is enough to encompass and encode higher level thinking. Empirically we observe that an LLM trained purely to predict a next token can do things like solve complex logic puzzles that it has never seen before. Skeptics claim that actually the network has seen at least analogous puzzles before and all it is doing is translating between them. However the novelty of what can be solved is very surprising. Intuitively it makes sense that at some level, that intelligence itself becomes a compression algorithm. For example, you can learn separately how to solve every puzzle ever presented to mankind, but that would take a lot of space. At some point it's more efficient to just learn "intelligence" itself and then apply that to the problem of predicting the next token. Once you do that you can stop trying to store an infinite database of parallel heuristics and just focus the parameter space on learning "common heuristics" that apply broadly across the problem space, and then apply that to every problem. The question is, at what parameter count and volume of training data does the situation flip to favoring "learning intelligence" rather than storing redundant domain specialised heuristics? And is it really happening? I would have thought just looking at the activation patterns could tell you a lot, because if common activations happen for entirely different problem spaces then you can argue that the network has to be learning common abstractions. If not, maybe it's just doing really large scale redundant storage of heurstics.
- disambiguation 1y agoGood take, but while we're invoking intuition, something is clearly missing in the fundamental design given real brains don't need to consume all the worlds literature before demonstrating intelligence. There's some missing piece w.r.t self learning and sense making. The path to emergent reasoning you lay out is interesting and might happen anyway as we scale up, but the original idea was to model these algorithms in our own image in the first place - I wonder if we won't discover that missing piece first.
- adampwells 1y ago> However the novelty of what can be solved is very surprising. I've read that the 'surprise' factor is much reduced when you actually see just how much data these things are trained on - far more than a human mind can possibly hold and (almost) endlessly varied. I.e. there is 'probably' something in the training set close to what 'surprised' you.
- eutropia 1y agoI often find that people using the word emergent to describe properties of a system tend to ascribe quasi magical properties to the system. Things tend to get vague and hand wavy when that term comes up. Just call them properties with unknown provenance.
- gond 1y ago> Just call them properties with unknown provenance. They would if it would be the correct designation, however, it is not. Emergence does not equal non-understanding or some spooky-hooky force coming from the unknown.Reductionism does not lead to an explaining-away of emergence.
- trash_cat 1y agoI think the better question is to answer why do emergent properties exist in the first place. I disagree with the premise that emergence is binary. It's not. What we determine "emergent behaviour" is partly a social concept. We decide when an LLM is good enough for us and when it "solved" something through emergent properties.