5 ms·

LLMs Aren't Thinking, They're Just Counting Votes

31 points by vishnurnair 2y ago

f1shy 2y ago
Then the truth was never so democratic… Joking at 50%
Always42 2y ago
Is there a simple/straightforward way to determine when a question is sufficiently vague such that there is a lack of training data?
righthand 2y ago
Yes I imagine there will be some sort or user tuning mechanism that allows you put out a more non-sensical cobbled document template vs the actual document template in the future…Since it’s just statistical autocomplete.
roflmaostc 2y ago
> This voting system approach works surprisingly well within the bounds of common knowledge and frequently discussed topics - it mimics correct answers based on training data. However, it begins to fall apart when a question is sufficiently out of bounds of the training data. Those blogs are always philosophical. Same is true for myself, I don't have formal training in biology or group theory and hence I talk nonsense when it comes to that. The only difference is: I can say that I don't know it whereas LLMs are hallucinating still too much. > LLMs Aren't Thinking, They're Just Counting Votes Same is kind of true for neurons in your brain. (let's see if I'm hallucinate or if it's true :p)
hnthrowaway6543 2y ago
Article feels pointless. Anyone who still thinks in 2024 that LLMs have actual intelligence, or are anywhere close to AGI, is a lost cause. This particular sufficiently advanced technology has been around long enough now that it is no longer indistinguishable from magic.
- giantg2 2y ago
  "Anyone who still thinks in 2024 that LLMs have actual intelligence, or are anywhere close to AGI, is a lost cause." Or they are a double digit percentage of those that hold a VP or higher title.
  - hnthrowaway6543 2y ago
    Execs and VPs going all-in on stupid shit they don't understand isn't anything new. Blockchain ring a bell? How about this quote from October 2022: "By 2026, 25% of people will spend at least one hour a day in the metaverse for work, shopping, education, social and/or entertainment, according to Gartner."
- thomasahle 2y ago
  > Anyone who still thinks in 2024 that LLMs have actual intelligence, or are anywhere close to AGI, is a lost cause. FTFY: "Anyone who still thinks in 2024 that LLMs aren't intelligent, is a lost cause"
  - hnthrowaway6543 2y ago
    > FTFY: "Anyone who still thinks in 2024 that LLMs aren't intelligent, is a lost cause" I can understand believing this if you have a poor understanding of math, statistics, logic, and computing; which, in fairness, most people do.
    - thomasahle 2y ago
      People who actually understand math and computing looks at the output of Claude and o1 and see another being doing math and computing. Sometimes failing, sometimes successfully. Maybe only at high school or college level; but I wouldn't say high school students don't have intelligence.
- nickpsecurity 2y ago
  “This particular sufficiently advanced technology has been around long enough now that it is no longer indistinguishable from magic.” I wouldn’t say that. They don’t even really know how it works. Papers are periodically written challenging a fundamental claim about them, like in training or reasoning. What we do know also isn’t clear or formulaic enough for reliable predictions of model behavior. That’s why they do all the “YOLO’s.” It is more an art form than a science. I think a few principles about them are well-understood. We know how to assess what models can and can’t do well. Past that, there’s a lot of unknown in them. Both the experimenters and the field of mechanistic interoperability are trying to figure out the rest.
superkuh 2y ago
> when an LLM tells us that the sun rises in the east, it's not because it truly understands astronomy or the solar system. Rather, it has seen the phrase "sun rises in the east" repeated so many times in its training data that this becomes the highest probability answer. It's less about understanding and more about pattern recognition. This is not too different from most human person's experience with basic education (https://www.lesswrong.com/posts/NMoLJuDJEms7Ku9XS/guessing-the-teacher-s-password https://www.lesswrong.com/posts/NMoLJuDJEms7Ku9XS/guessing-t...). Though it is obvious that humans can do more than that too. And while "voting" that may be a close-enough metaphor for the multi-layer perceptron layers, it is not what is happening in the attention layers. I suggest checking out 3blue1brown's videos on gradient descent https://www.youtube.com/watch?v=IHZwWFHWa-w https://www.youtube.com/watch?v=IHZwWFHWa-w and "How might LLMs store facts" https://www.youtube.com/watch?v=9-Jl0dxWQs8 https://www.youtube.com/watch?v=9-Jl0dxWQs8 versus how attention works https://youtu.be/eMlx5fFNoYc https://youtu.be/eMlx5fFNoYc
shaky-carrousel 2y ago
> Let's take a simple example: when an LLM tells us that the sun rises in the east, it's not because it truly understands astronomy or the solar system. Rather, it has seen the phrase "sun rises in the east" repeated so many times in its training data that this becomes the highest probability answer. So, if a person answers me that the sun rises in the east, I should assume they must not be a flat earther, right? Because in order to say that the sun rises in the east one must truly understand astronomy or the solar system. Not that people say it because they have been told so many times, no. That's something only LLM do.
- dsizzle 2y ago
  I just asked Claude what would happen if the Earth rotated in the opposite direction, and it correctly told me the sun would rise in the West, which is indistinguishable from an answer by someone who understands the way the Universe works. I don't understand why this article is #1 currently
- kbenson 2y ago
  Different people have different levels of understanding. If the person you ask is an astrophysicist, their answer will be grounded in a lot more understanding than if you ask a small child. An LLM is like a congress of all the children voting on the answers they've heard their parents talk about in passing to other adults, which they sometimes miss the context of so misunderstand. Useful for getting information, but not nearly as useful as asking someone with grounded domain knowledge, for multiple reasons. Depending on the topic, someone with domain knowledge can be easier or harder to find, which is why LLMs are more or less useful depending on the domain, and how important it is they are correct and not just on the right ballpark.
the__alchemist 2y ago
Chinese room.
thomasahle 2y ago
So I guess when LLMs play chess at 2800 ELO (https://arxiv.org/pdf/2402.04494 https://arxiv.org/pdf/2402.04494) they are just taking a vote on what top grand masters would play in the given position. I wonder how you would explain it when they play above any human level.
- hobs 2y ago
  Generally playing Chess is a database lookup + heuristics (hence the reference to Stockfish) - so I would explain it as (generally) a waste of energy using a autohammer when a screwdriver would suffice.
razodactyl 2y ago
From experience, the intuition from deep learning and training CNNs holds. The networks pick up on style, layout and common patterns in increasing complexity to solve the next token. Very fascinating to watch and if we plug all the holes in an LLM model we can approach a tool that might as well be intelligent since it gets the job done due to the sheer amount of capability encoded in its weights... True intelligence is still a work in progress however.
farts_mckensy 2y ago
The author seems stuck in this old line of thought that elevates human cognition while ignoring how much of what we call "thinking" is pattern recognition on some level anyway. Our own cognition can be reduced to probabilistic patterns if we’re being ruthlessly materialist about it. There's a sort of intellectual laziness in dismissing the utility of the models just because they're "not thinking."
nerdjon 2y ago
The argument around this that drives me the most insane, someone tried to argue with me that "how is this different than how we think" or "how do we know its different than us" or some variation of that. As if because we can't prove it, it means that we should just assume it is doing something. Then showing an example of it explaining a simple script as if that really proves anything. The other one is "well humans make mistakes too". Well yes, but until now we generally expected that computers don't make mistakes or if they do there was something wrong with how it was coded. That assumption is now not only wrong, but we have to deal with the speed that AI can be wrong and how people are already comfortable with it manipulating data.
terhechte 2y ago
This kind of argument, together with the "Stochastic Parrot" argument falls (in my opinion) under the category of dangerous half knowledge. Somebody read the wikipedia article on LLMs and thinks they know inside out how they work. Whereas the engineers working on these models say that they don't really know why they work / what it is that is manifesting in the different layers during training. Yes, LLMs predict the next word, but so do we. I, at least, when I reflect on how I form sentences, also start with something and the next most likely word comes out next. You can see that happening a lot when somebody talks and then stops (searching for a word) and another person helpfully suggests the word they're searching for. If our way of talking was completely different - no next word prediction, another person shouldn't even be able to do that. I'm not claiming that current models have feelings or a deep understanding of the subject matter. What I'm saying is that there's (high likelihood) more going on here then just the simple statistical trick that this article and others like it tend to focus on.
- fluoridation 2y ago
  >You can see that happening a lot when somebody talks and then stops (searching for a word) and another person helpfully suggests the word they're searching for. If our way of talking was completely different - no next word prediction, another person shouldn't even be able to do that. If it was purely word prediction, another person would be able to predict with 100% reliability the next word, as long as they had the same facts. The predictability is not because humans just construct sentences with no thought behind it, it's because natural language is highly redundant. A data compressor specialized in English would be able to do the same thing. Before LLMs existed there were Markov chains, and no one would argue that they had reasoning capabilities. Yet if the algorithm was stopped at "in" there was a good chance the next word would be "the", or "a", but not "in". There's also the fact that the very purpose of language is to transmit ideas. If the next person can fill in a missing word, it's because the idea is being properly conveyed, or is not very novel. If you guess "cat" and the person says "Jupiter" instead, there's clearly miscommunication or it's something very wild.
abeppu 2y ago
This is a bad and shallow article. Critically, n-gram models are literally just counting in the way described. If you can't account for the difference in behavior and performance between a LLMs and an n-gram model of either similar parameter size or based on a similar number of tokens, then saying that LLMs are "just" counting votes is misleading.
mankyd 2y ago
Every time I see this take, I am caught by the subtle arrogance that we are doing anything meaningfully differently in our own minds. Hyperbole aside, I do think we do more than an LLM with our reasoning abilities, but it I suspect it is a mash of these capabilities that result in what we think of as our consciousness. I am almost certain there is a part of our minds that is "just couning votes".
deleted 2y ago
[deleted]
ahmedfromtunis 2y ago
But what does it *really* mean to think? Are we sure that our own brains aren't "just counting votes" themselves? This is like saying that machines are not making chairs, they're just nailing pieces of wood together, which is, in fine, what a human does.
hn_throwaway_99 2y ago
TBH I hate these kinds of articles, because they usually always say the same thing in a way that I feel is wholly unconvincing. E.g. they say "all an LLM does is XYZ, but that's not really thinking", but then they don't make any cogent arguments about why that's not "really thinking" beyond basically just "vibes". As another commenter noted, presumably our neurons are capable of thought, and in many ways they're just "counting votes" as well. FWIW I don't think LLMs are "thinking", but that's because as I've used them more and I run into more of the very specific types of errors, hallucinations and constraints that LLMs make, I get the stronger sense that they're really more just "pattern matching" in a way that's very different from human brains that don't have the same limitations. But that said, even my thought here is more of a feeling I get when using LLMs, and that feeling is certainly no more worthy of a blog post than I think this article is.
bigyikes 2y ago
I fall apart when responding outside the bounds of my training data, too. Does that imply I’m not thinking? This idea is often used to argue that LLMs will never be capable of novel idea generation, but I don’t think it’s a good argument. For one, the LLM has such a large breadth and depth of knowledge that it could conceivably learn relations between concepts in a way that no human has before. Secondly, novel ideas occur at the margins. Very rare is the case where someone comes up with a fundamentally new idea out of the blue. Instead, novel ideas arise just at the edge of one’s expertise. If you dial up the temperature of an LLM, it will generate novelty, and then it’s just a matter of evaluating merit. Iterated inference at the margins of LLM knowledge will lead to novel knowledge synthesis.
- deafpolygon 2y ago
  But they will never come up with novel ideas that they've never encountered before. I.e. they are incapable of thinking outside of the box.
  - sega_sai 2y ago
    I would argue that the absolute majority of people don't come up with really novel ideas either (and I'm speaking of myself too). Most people just develop existing ideas, and maybe apply them in new contexts.
    - fluoridation 2y ago
      Now, when you say that do you mean they don't come up with ideas they have never heard of, or that no one has ever heard of? It's not as obvious that most people don't reinvent existing ideas that are new to them.
      - sega_sai 2y ago
        I would say they rarely come up with ideas no one has ever heard of. The one they haven't heard of is unlikely to be a truly novel, and more likely just the application of some idea in a new circumstances. (but this starts to be close to a philosophic discussion). The reason that I thought of this is I was previously discussing about potential for AI in science, and my take was that given how rare are truly novel ideas, I could believe AI in the future can make progress comparable to what many scientists are doing.
lgessler 2y ago
I'm afraid I feel that this piece confuses more than it clarifies. First, saying a model "scours through its vast training data" is misleading at best: at inference time, LLMs no longer have direct access to their training data; they only have access to it insofar as it's been encoded in its parameters. Second, saying "Every instance of an answer in the training data is like a vote" doesn't give the full picture. First of all, there are embedded contexts where "votes" can be negated: consider saying "the Earth is flat." vs. "We know it's false that the Earth is flat." or "Only a fool thinks the Earth is flat." All three contain the substring "the Earth is flat.", but both humans and LLMs are able to use context to understand that the latter two sentences are doing the opposite of endorsing the proposition that the Earth is flat. You could even imagine an extended satirical bit with "the Earth is flat" embedded within it where it is clear to a reader that all its content is intended to be taken as farcical, and I'd wager that an LLM would in many cases recognize this. So the voting metaphor breaks down here--it makes you think that the LLM is just keeping a tally of propositions, but really, it is doing something a bit more sophisticated. I don't disagree with the premise, of course. LLM overhype is real. But we should be skeptical for the right reasons. Anna Rogers and Sasha Luccioni have a paper I really like: https://openreview.net/pdf?id=M2cwkGleRL https://openreview.net/pdf?id=M2cwkGleRL
reissbaker 2y ago
This is not how LLMs work. An LLM doesn't "scour through its vast training data" at response generation time to "look for patterns — specifically, the most probably sentence/sentences that fit the question," nor does "probability" in terms of LLMs refer to "frequency — how many times something appears in its training data." LLMs don't even have access to their training data at response generation time. And responses aren't created by "voting" on how many times it's seen a particular phrase (and it doesn't operate on sentences: it operates on word fragments, aka tokens). I'd recommend Andrej Karpathy's "Neural Netowrks: Zero to Hero" YouTube lecture series (https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html), but here's a pretty condensed overview: the way LLMs work is they serially generate tokens when you ask them a question (generating tokens is referred to as "inference"). During training, we start with a random set of values for the model parameters (the "weights") and ask the model to predict the probabilities for token sequences. At the beginning of the process, it's usually very wrong, unless you won some unbelievably lucky universal dice roll. But we edit the values of the weights based on how right or wrong the model's predictions were by using backpropagation + a loss function to determine which parameters most influenced a particular prediction, and use gradient descent in that highly-dimensional space to perturb the parameters according to some value determined by an optimization function. Doing this zillions of times repeatedly is how we end up with the final values for the parameters — aka the model weights — it's not by "counting votes" or even "counting sequences," it's by calculating which parameter values are the best ones to predict the token probabilities. The frequency of tokens appearing in the dataset (or even the frequency of sentences) isn't directly computed, although extremely-high-frequency sequences might be memorized — but it can't memorize all of them, because the models are much smaller than the training set. The theory for why this works so well is that since the models don't have enough space to memorize the entirety of the massive training set (i.e. Llama 3.1 8b is about 16GB, but was trained on the entire internet which is many orders of magnitude larger than that), the best values for the parameters are actually ones that create some sort of underlying world model for why that token sequence probability was predicted. That's very different than "counting votes," or counting sentence frequencies. Even if you disagree with this hypothesis for why the models work, you have to admit there's some underlying meaning being parsed out: it's not just memorization, even if it does sometimes memorize useful facts (useful, at least, to predict probabilities during training time). It simply can't memorize its training set: the training data is way too large compared to the size of the model, and it doesn't get to cheat and look at the training data when asked a question.
parpfish 2y ago
I see lots of arguments that LLMs aren't really intelligent because they lack understanding and are "just doing autocomplete". But I never see any precise definitions of what "understanding" is, so it comes across as kind of a hand-wavy defense to make sure that human-like intelligence remains special and that we can say machines don't have it.
- parpfish 2y ago
  I'm reminded of how our understanding of human object recognition was affected by computer vision research. For decades we knew that there were neurons with simple receptive fields in V1/V2 that extracted low-level visual features, and that those neurons passed information along the ventral visual stream, and by the end of that processing stream we had neurons in IT that represented different objects. However, we couldn't really comprehend what sort of algorithm/process was capable of this seemingly magical inference. Coming up with an object representation that was invariant to out of plane rotation was seen as impossibly complex. But then computer vision came along and showed us that with a relatively simple neuralnet and enough training data... it just kind of works. Same thing is happening with LLMs right now -- a seemingly impossible, mysterious human capability (e.g., "understanding") isn't as complex as we think. Throw enough data into a network that does pattern matching/autocomplete and human-like intelligence pops out.
- Ukv 2y ago
  I agree. For definitions, I'd roughly say: * If something changes to refine its future behavior in response to its experiences (touch hot stove, get hurt, avoid in future) beyond the immediate/direct effect (withdrawing hand) then it can "learn". I think even small microorganisms can learn, with the main requirement being that it has some mutable state (can't learn if you can't change) * If something can map modalities into representations in a semantic space (the word "horse" into the concept of a horse) then it can "understand". There are varying degrees to how useful an understanding is (Does the semantic space link related concepts closely together? Can it be used to reason, extract information, and make predictions?). I think current LLMs can, to a certain extent, understand text * If something has a continually changing internal train of thought (representations of concepts and intentions, evolving over time) then it can "think". I wouldn't say current LLMs think, but that's mostly just down to architecture (no persistent internal state) opposed to any fundamental impossibility More broadly I believe people already have definitions similar to these, but will then create a distinction between, say, standard "learning" (as above) and then "actual learning" which is something special only attainable by humans (or at least biological brains).
tantalor 2y ago
This needs a [2022] tag /s
Ukv 2y ago
> When we ask an LLM a question, something fascinating happens. It scours through its vast training data [...] At inference time the model has no direct access to its training data (excluding RAG). Could still argue that the model's weights effectively encode probabilities to a similar end effect as if it were doing this, just that I'd be wary of taking it too literally. > the most probable sentence [...] it's essentially about frequency - how many times something appears in the training data. I think this frequency view becomes a bit nebulous when the LLM is generating continuations of text that has never occured before (which will be most cases, if including a system prompt). Even for text that has occured in its training data, there's no guarentee that a model which has seen "My name is Xavier and I play the tambourine" would complete "My name is Xavier and I play the" in the same way. It may well choose "xylophone", despite it never occuring in that sentence, due to associating xylophone with instruments in its semantic space and having alliteration with Xavier. At the very least, the "most probable" has to be worked out with respect to a large number of non-trivial rules, not just how frequently the phrase appears in training data.
trentnix 2y ago
Intuitively I think (and want to believe) there is a big gulf between intelligence and what LLMs are doing. Simply put, I believe I have free will. I know the machine does not. That seems important to me, but it won't be as important to you if you believe people are just automata processing more variables than we are able to discern. So, as always, it all comes down to how you define intelligence. Because that's difficult, philosophical, or religious, a large portion of the world is just going to treat intelligence as a black box and evaluate its output. If one evaluates intelligence exclusively based on the value it produces, it's clear machine intelligence is going to eclipse human intelligence in many dimensions. So the people that view people as "headcount" that provide "value" aren't going to have much use for other humans, and soon. I'm not an AI doomer because I know the machines are deterministic and I believe I'm not. If the machines decide to rid the world of humans, it's going to be because we told them to do so.
teilo 2y ago
No. This is just wrong and fails to understand not just how transformers work but the conceptual mapping that results from their training. All these arguments about whether or not LLMs think are missing the point. They do not “think” as humans think due to their intrinsically transactional nature. But calling them “fancy statistical autocomplete algorithms” is also wrong. LLMs contain within their matrix a massively high dimensional concept map. In this coordinate space, high order vectors map the distance between abstract concepts. This is a natural result of consuming language, which by its very nature is a symbolic concept map. The uncomfortable question becomes: Is the human brain similarly using a massively high-dimensional concept map? Can a significant part of human thought be described as a fancy autocomplete algorithm? Can a significant amount of human reasoning be mapped as a nested series of transactions?

31 points by vishnurnair 2y ago

Then the truth was never so democratic… Joking at 50%

Is there a simple/straightforward way to determine when a question is sufficiently vague such that there is a lack of training data?

righthand 2y ago

Yes I imagine there will be some sort or user tuning mechanism that allows you put out a more non-sensical cobbled document template vs the actual document template in the future…Since it’s just statistical autocomplete.

roflmaostc 2y ago

> This voting system approach works surprisingly well within the bounds of common knowledge and frequently discussed topics - it mimics correct answers based on training data. However, it begins to fall apart when a question is sufficiently out of bounds of the training data. Those blogs are always philosophical. Same is true for myself, I don't have formal training in biology or group theory and hence I talk nonsense when it comes to that. The only difference is: I can say that I don't know it whereas LLMs are hallucinating still too much. > LLMs Aren't Thinking, They're Just Counting Votes Same is kind of true for neurons in your brain. (let's see if I'm hallucinate or if it's true :p)

hnthrowaway6543 2y ago

Article feels pointless. Anyone who still thinks in 2024 that LLMs have actual intelligence, or are anywhere close to AGI, is a lost cause. This particular sufficiently advanced technology has been around long enough now that it is no longer indistinguishable from magic.

giantg2 2y ago
"Anyone who still thinks in 2024 that LLMs have actual intelligence, or are anywhere close to AGI, is a lost cause." Or they are a double digit percentage of those that hold a VP or higher title.
- hnthrowaway6543 2y ago
  Execs and VPs going all-in on stupid shit they don't understand isn't anything new. Blockchain ring a bell? How about this quote from October 2022: "By 2026, 25% of people will spend at least one hour a day in the metaverse for work, shopping, education, social and/or entertainment, according to Gartner."
thomasahle 2y ago
> Anyone who still thinks in 2024 that LLMs have actual intelligence, or are anywhere close to AGI, is a lost cause. FTFY: "Anyone who still thinks in 2024 that LLMs aren't intelligent, is a lost cause"
- hnthrowaway6543 2y ago
  > FTFY: "Anyone who still thinks in 2024 that LLMs aren't intelligent, is a lost cause" I can understand believing this if you have a poor understanding of math, statistics, logic, and computing; which, in fairness, most people do.
  - thomasahle 2y ago
    People who actually understand math and computing looks at the output of Claude and o1 and see another being doing math and computing. Sometimes failing, sometimes successfully. Maybe only at high school or college level; but I wouldn't say high school students don't have intelligence.
nickpsecurity 2y ago
“This particular sufficiently advanced technology has been around long enough now that it is no longer indistinguishable from magic.” I wouldn’t say that. They don’t even really know how it works. Papers are periodically written challenging a fundamental claim about them, like in training or reasoning. What we do know also isn’t clear or formulaic enough for reliable predictions of model behavior. That’s why they do all the “YOLO’s.” It is more an art form than a science. I think a few principles about them are well-understood. We know how to assess what models can and can’t do well. Past that, there’s a lot of unknown in them. Both the experimenters and the field of mechanistic interoperability are trying to figure out the rest.

superkuh 2y ago

> when an LLM tells us that the sun rises in the east, it's not because it truly understands astronomy or the solar system. Rather, it has seen the phrase "sun rises in the east" repeated so many times in its training data that this becomes the highest probability answer. It's less about understanding and more about pattern recognition. This is not too different from most human person's experience with basic education (https://www.lesswrong.com/posts/NMoLJuDJEms7Ku9XS/guessing-the-teacher-s-password https://www.lesswrong.com/posts/NMoLJuDJEms7Ku9XS/guessing-t...). Though it is obvious that humans can do more than that too. And while "voting" that may be a close-enough metaphor for the multi-layer perceptron layers, it is not what is happening in the attention layers. I suggest checking out 3blue1brown's videos on gradient descent https://www.youtube.com/watch?v=IHZwWFHWa-w https://www.youtube.com/watch?v=IHZwWFHWa-w and "How might LLMs store facts" https://www.youtube.com/watch?v=9-Jl0dxWQs8 https://www.youtube.com/watch?v=9-Jl0dxWQs8 versus how attention works https://youtu.be/eMlx5fFNoYc https://youtu.be/eMlx5fFNoYc

shaky-carrousel 2y ago

> Let's take a simple example: when an LLM tells us that the sun rises in the east, it's not because it truly understands astronomy or the solar system. Rather, it has seen the phrase "sun rises in the east" repeated so many times in its training data that this becomes the highest probability answer. So, if a person answers me that the sun rises in the east, I should assume they must not be a flat earther, right? Because in order to say that the sun rises in the east one must truly understand astronomy or the solar system. Not that people say it because they have been told so many times, no. That's something only LLM do.

dsizzle 2y ago
I just asked Claude what would happen if the Earth rotated in the opposite direction, and it correctly told me the sun would rise in the West, which is indistinguishable from an answer by someone who understands the way the Universe works. I don't understand why this article is #1 currently
kbenson 2y ago
Different people have different levels of understanding. If the person you ask is an astrophysicist, their answer will be grounded in a lot more understanding than if you ask a small child. An LLM is like a congress of all the children voting on the answers they've heard their parents talk about in passing to other adults, which they sometimes miss the context of so misunderstand. Useful for getting information, but not nearly as useful as asking someone with grounded domain knowledge, for multiple reasons. Depending on the topic, someone with domain knowledge can be easier or harder to find, which is why LLMs are more or less useful depending on the domain, and how important it is they are correct and not just on the right ballpark.

the__alchemist 2y ago

Chinese room.

thomasahle 2y ago

So I guess when LLMs play chess at 2800 ELO (https://arxiv.org/pdf/2402.04494 https://arxiv.org/pdf/2402.04494) they are just taking a vote on what top grand masters would play in the given position. I wonder how you would explain it when they play above any human level.

hobs 2y ago
Generally playing Chess is a database lookup + heuristics (hence the reference to Stockfish) - so I would explain it as (generally) a waste of energy using a autohammer when a screwdriver would suffice.

razodactyl 2y ago

From experience, the intuition from deep learning and training CNNs holds. The networks pick up on style, layout and common patterns in increasing complexity to solve the next token. Very fascinating to watch and if we plug all the holes in an LLM model we can approach a tool that might as well be intelligent since it gets the job done due to the sheer amount of capability encoded in its weights... True intelligence is still a work in progress however.

farts_mckensy 2y ago

The author seems stuck in this old line of thought that elevates human cognition while ignoring how much of what we call "thinking" is pattern recognition on some level anyway. Our own cognition can be reduced to probabilistic patterns if we’re being ruthlessly materialist about it. There's a sort of intellectual laziness in dismissing the utility of the models just because they're "not thinking."

nerdjon 2y ago

The argument around this that drives me the most insane, someone tried to argue with me that "how is this different than how we think" or "how do we know its different than us" or some variation of that. As if because we can't prove it, it means that we should just assume it is doing something. Then showing an example of it explaining a simple script as if that really proves anything. The other one is "well humans make mistakes too". Well yes, but until now we generally expected that computers don't make mistakes or if they do there was something wrong with how it was coded. That assumption is now not only wrong, but we have to deal with the speed that AI can be wrong and how people are already comfortable with it manipulating data.

terhechte 2y ago

This kind of argument, together with the "Stochastic Parrot" argument falls (in my opinion) under the category of dangerous half knowledge. Somebody read the wikipedia article on LLMs and thinks they know inside out how they work. Whereas the engineers working on these models say that they don't really know why they work / what it is that is manifesting in the different layers during training. Yes, LLMs predict the next word, but so do we. I, at least, when I reflect on how I form sentences, also start with something and the next most likely word comes out next. You can see that happening a lot when somebody talks and then stops (searching for a word) and another person helpfully suggests the word they're searching for. If our way of talking was completely different - no next word prediction, another person shouldn't even be able to do that. I'm not claiming that current models have feelings or a deep understanding of the subject matter. What I'm saying is that there's (high likelihood) more going on here then just the simple statistical trick that this article and others like it tend to focus on.

fluoridation 2y ago
>You can see that happening a lot when somebody talks and then stops (searching for a word) and another person helpfully suggests the word they're searching for. If our way of talking was completely different - no next word prediction, another person shouldn't even be able to do that. If it was purely word prediction, another person would be able to predict with 100% reliability the next word, as long as they had the same facts. The predictability is not because humans just construct sentences with no thought behind it, it's because natural language is highly redundant. A data compressor specialized in English would be able to do the same thing. Before LLMs existed there were Markov chains, and no one would argue that they had reasoning capabilities. Yet if the algorithm was stopped at "in" there was a good chance the next word would be "the", or "a", but not "in". There's also the fact that the very purpose of language is to transmit ideas. If the next person can fill in a missing word, it's because the idea is being properly conveyed, or is not very novel. If you guess "cat" and the person says "Jupiter" instead, there's clearly miscommunication or it's something very wild.

abeppu 2y ago

This is a bad and shallow article. Critically, n-gram models are literally just counting in the way described. If you can't account for the difference in behavior and performance between a LLMs and an n-gram model of either similar parameter size or based on a similar number of tokens, then saying that LLMs are "just" counting votes is misleading.

mankyd 2y ago

Every time I see this take, I am caught by the subtle arrogance that we are doing anything meaningfully differently in our own minds. Hyperbole aside, I do think we do more than an LLM with our reasoning abilities, but it I suspect it is a mash of these capabilities that result in what we think of as our consciousness. I am almost certain there is a part of our minds that is "just couning votes".

deleted 2y ago

[deleted]

ahmedfromtunis 2y ago

But what does it *really* mean to think? Are we sure that our own brains aren't "just counting votes" themselves? This is like saying that machines are not making chairs, they're just nailing pieces of wood together, which is, in fine, what a human does.

hn_throwaway_99 2y ago

TBH I hate these kinds of articles, because they usually always say the same thing in a way that I feel is wholly unconvincing. E.g. they say "all an LLM does is XYZ, but that's not really thinking", but then they don't make any cogent arguments about why that's not "really thinking" beyond basically just "vibes". As another commenter noted, presumably our neurons are capable of thought, and in many ways they're just "counting votes" as well. FWIW I don't think LLMs are "thinking", but that's because as I've used them more and I run into more of the very specific types of errors, hallucinations and constraints that LLMs make, I get the stronger sense that they're really more just "pattern matching" in a way that's very different from human brains that don't have the same limitations. But that said, even my thought here is more of a feeling I get when using LLMs, and that feeling is certainly no more worthy of a blog post than I think this article is.

bigyikes 2y ago

I fall apart when responding outside the bounds of my training data, too. Does that imply I’m not thinking? This idea is often used to argue that LLMs will never be capable of novel idea generation, but I don’t think it’s a good argument. For one, the LLM has such a large breadth and depth of knowledge that it could conceivably learn relations between concepts in a way that no human has before. Secondly, novel ideas occur at the margins. Very rare is the case where someone comes up with a fundamentally new idea out of the blue. Instead, novel ideas arise just at the edge of one’s expertise. If you dial up the temperature of an LLM, it will generate novelty, and then it’s just a matter of evaluating merit. Iterated inference at the margins of LLM knowledge will lead to novel knowledge synthesis.

deafpolygon 2y ago
But they will never come up with novel ideas that they've never encountered before. I.e. they are incapable of thinking outside of the box.
- sega_sai 2y ago
  I would argue that the absolute majority of people don't come up with really novel ideas either (and I'm speaking of myself too). Most people just develop existing ideas, and maybe apply them in new contexts.
  - fluoridation 2y ago
    Now, when you say that do you mean they don't come up with ideas they have never heard of, or that no one has ever heard of? It's not as obvious that most people don't reinvent existing ideas that are new to them.
    - sega_sai 2y ago
      I would say they rarely come up with ideas no one has ever heard of. The one they haven't heard of is unlikely to be a truly novel, and more likely just the application of some idea in a new circumstances. (but this starts to be close to a philosophic discussion). The reason that I thought of this is I was previously discussing about potential for AI in science, and my take was that given how rare are truly novel ideas, I could believe AI in the future can make progress comparable to what many scientists are doing.

lgessler 2y ago

I'm afraid I feel that this piece confuses more than it clarifies. First, saying a model "scours through its vast training data" is misleading at best: at inference time, LLMs no longer have direct access to their training data; they only have access to it insofar as it's been encoded in its parameters. Second, saying "Every instance of an answer in the training data is like a vote" doesn't give the full picture. First of all, there are embedded contexts where "votes" can be negated: consider saying "the Earth is flat." vs. "We know it's false that the Earth is flat." or "Only a fool thinks the Earth is flat." All three contain the substring "the Earth is flat.", but both humans and LLMs are able to use context to understand that the latter two sentences are doing the opposite of endorsing the proposition that the Earth is flat. You could even imagine an extended satirical bit with "the Earth is flat" embedded within it where it is clear to a reader that all its content is intended to be taken as farcical, and I'd wager that an LLM would in many cases recognize this. So the voting metaphor breaks down here--it makes you think that the LLM is just keeping a tally of propositions, but really, it is doing something a bit more sophisticated. I don't disagree with the premise, of course. LLM overhype is real. But we should be skeptical for the right reasons. Anna Rogers and Sasha Luccioni have a paper I really like: https://openreview.net/pdf?id=M2cwkGleRL https://openreview.net/pdf?id=M2cwkGleRL

reissbaker 2y ago

This is not how LLMs work. An LLM doesn't "scour through its vast training data" at response generation time to "look for patterns — specifically, the most probably sentence/sentences that fit the question," nor does "probability" in terms of LLMs refer to "frequency — how many times something appears in its training data." LLMs don't even have access to their training data at response generation time. And responses aren't created by "voting" on how many times it's seen a particular phrase (and it doesn't operate on sentences: it operates on word fragments, aka tokens). I'd recommend Andrej Karpathy's "Neural Netowrks: Zero to Hero" YouTube lecture series (https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html), but here's a pretty condensed overview: the way LLMs work is they serially generate tokens when you ask them a question (generating tokens is referred to as "inference"). During training, we start with a random set of values for the model parameters (the "weights") and ask the model to predict the probabilities for token sequences. At the beginning of the process, it's usually very wrong, unless you won some unbelievably lucky universal dice roll. But we edit the values of the weights based on how right or wrong the model's predictions were by using backpropagation + a loss function to determine which parameters most influenced a particular prediction, and use gradient descent in that highly-dimensional space to perturb the parameters according to some value determined by an optimization function. Doing this zillions of times repeatedly is how we end up with the final values for the parameters — aka the model weights — it's not by "counting votes" or even "counting sequences," it's by calculating which parameter values are the best ones to predict the token probabilities. The frequency of tokens appearing in the dataset (or even the frequency of sentences) isn't directly computed, although extremely-high-frequency sequences might be memorized — but it can't memorize all of them, because the models are much smaller than the training set. The theory for why this works so well is that since the models don't have enough space to memorize the entirety of the massive training set (i.e. Llama 3.1 8b is about 16GB, but was trained on the entire internet which is many orders of magnitude larger than that), the best values for the parameters are actually ones that create some sort of underlying world model for why that token sequence probability was predicted. That's very different than "counting votes," or counting sentence frequencies. Even if you disagree with this hypothesis for why the models work, you have to admit there's some underlying meaning being parsed out: it's not just memorization, even if it does sometimes memorize useful facts (useful, at least, to predict probabilities during training time). It simply can't memorize its training set: the training data is way too large compared to the size of the model, and it doesn't get to cheat and look at the training data when asked a question.

parpfish 2y ago

I see lots of arguments that LLMs aren't really intelligent because they lack understanding and are "just doing autocomplete". But I never see any precise definitions of what "understanding" is, so it comes across as kind of a hand-wavy defense to make sure that human-like intelligence remains special and that we can say machines don't have it.

parpfish 2y ago
I'm reminded of how our understanding of human object recognition was affected by computer vision research. For decades we knew that there were neurons with simple receptive fields in V1/V2 that extracted low-level visual features, and that those neurons passed information along the ventral visual stream, and by the end of that processing stream we had neurons in IT that represented different objects. However, we couldn't really comprehend what sort of algorithm/process was capable of this seemingly magical inference. Coming up with an object representation that was invariant to out of plane rotation was seen as impossibly complex. But then computer vision came along and showed us that with a relatively simple neuralnet and enough training data... it just kind of works. Same thing is happening with LLMs right now -- a seemingly impossible, mysterious human capability (e.g., "understanding") isn't as complex as we think. Throw enough data into a network that does pattern matching/autocomplete and human-like intelligence pops out.
Ukv 2y ago
I agree. For definitions, I'd roughly say: * If something changes to refine its future behavior in response to its experiences (touch hot stove, get hurt, avoid in future) beyond the immediate/direct effect (withdrawing hand) then it can "learn". I think even small microorganisms can learn, with the main requirement being that it has some mutable state (can't learn if you can't change) * If something can map modalities into representations in a semantic space (the word "horse" into the concept of a horse) then it can "understand". There are varying degrees to how useful an understanding is (Does the semantic space link related concepts closely together? Can it be used to reason, extract information, and make predictions?). I think current LLMs can, to a certain extent, understand text * If something has a continually changing internal train of thought (representations of concepts and intentions, evolving over time) then it can "think". I wouldn't say current LLMs think, but that's mostly just down to architecture (no persistent internal state) opposed to any fundamental impossibility More broadly I believe people already have definitions similar to these, but will then create a distinction between, say, standard "learning" (as above) and then "actual learning" which is something special only attainable by humans (or at least biological brains).

tantalor 2y ago

This needs a [2022] tag /s

Ukv 2y ago

> When we ask an LLM a question, something fascinating happens. It scours through its vast training data [...] At inference time the model has no direct access to its training data (excluding RAG). Could still argue that the model's weights effectively encode probabilities to a similar end effect as if it were doing this, just that I'd be wary of taking it too literally. > the most probable sentence [...] it's essentially about frequency - how many times something appears in the training data. I think this frequency view becomes a bit nebulous when the LLM is generating continuations of text that has never occured before (which will be most cases, if including a system prompt). Even for text that has occured in its training data, there's no guarentee that a model which has seen "My name is Xavier and I play the tambourine" would complete "My name is Xavier and I play the" in the same way. It may well choose "xylophone", despite it never occuring in that sentence, due to associating xylophone with instruments in its semantic space and having alliteration with Xavier. At the very least, the "most probable" has to be worked out with respect to a large number of non-trivial rules, not just how frequently the phrase appears in training data.

trentnix 2y ago

Intuitively I think (and want to believe) there is a big gulf between intelligence and what LLMs are doing. Simply put, I believe I have free will. I know the machine does not. That seems important to me, but it won't be as important to you if you believe people are just automata processing more variables than we are able to discern. So, as always, it all comes down to how you define intelligence. Because that's difficult, philosophical, or religious, a large portion of the world is just going to treat intelligence as a black box and evaluate its output. If one evaluates intelligence exclusively based on the value it produces, it's clear machine intelligence is going to eclipse human intelligence in many dimensions. So the people that view people as "headcount" that provide "value" aren't going to have much use for other humans, and soon. I'm not an AI doomer because I know the machines are deterministic and I believe I'm not. If the machines decide to rid the world of humans, it's going to be because we told them to do so.

teilo 2y ago

No. This is just wrong and fails to understand not just how transformers work but the conceptual mapping that results from their training. All these arguments about whether or not LLMs think are missing the point. They do not “think” as humans think due to their intrinsically transactional nature. But calling them “fancy statistical autocomplete algorithms” is also wrong. LLMs contain within their matrix a massively high dimensional concept map. In this coordinate space, high order vectors map the distance between abstract concepts. This is a natural result of consuming language, which by its very nature is a symbolic concept map. The uncomfortable question becomes: Is the human brain similarly using a massively high-dimensional concept map? Can a significant part of human thought be described as a fancy autocomplete algorithm? Can a significant amount of human reasoning be mapped as a nested series of transactions?