8 ms·

If commenters wish to know what is not, "guessing the next word", let me outline it. Compare, "I like what you were wearing", "Pass me the salt", and "Have you

If commenters wish to know what is not, "guessing the next word", let me outline it. Compare, "I like what you were wearing", "Pass me the salt", and "Have you been to London recently?" as generated by an LLM and as spoken by a person. What is the reason each piece of text (in a whatsapp chat, say) is provided? When the LLM generates each word it does so because it is, on average, the most common word in a corpus of text on which it was trained: "wearing" follows, "I like what you were" because most people who were having these conversations, captured in the training data, were talking about clothes. When a person types those words on a keyboard, the following are the causes: the speaker's mental states of recollection, preference, taste; the speaker's affective/attachement states with respect to their friend; the speaker's habitation into social cues; the speaker's imagining through recall what their friend was wearing; the speaker's ability to abstract from their memories into identifying clothing; and so on. Indeed, the cause of a person speaking is so vastly different to generating a word based on a historical frequency, that to suppose these are related seems incomprehensible. The only reason the illusion of similarity is effective is because the training data is a text-based observation of the causal process in people: the training data is distributed by people talking (and so on). Insofar as you cannot just replay variations on these prior conversations, the LLM will fail and expose itself as actually insensitive to any of these things. I'd encourage credulous fans of AI not to dehumanize themselves and others by the supposition that they speak because they are selecting an optimal word from a dictionary based on all prior conversations they were a part of. You aren't doing that.

jameshart 2y ago
LLMs don’t ’replay variations on prior conversations’ though. Predicting the probable token in a conversation requires predicting the probable subject of the conversation, predicting the interlocutors’ relationship and manner of speaking to one another, predicting the state of recollection, preference and taste of the speaker, predicting the speaker’s mental model… If the LLM isn’t predicting all of those things then it will produce poor predictions of the next word; doing it well - and humans tend to agree that in a vast array of cases state of the art LLMs do predict tokens very well - requires that prediction model to predict all that context as well.
- mjburgess 2y ago
  Alas, no it doesn't. Language induces this sort of anthropomorphism in people, I guess, so consider images. Suppose I take a billion images of all the coffee cups in the world, at a set of angles on the cup, and then build an associative (ie., frequency) statistical model of their pixels (ie., statistical AI). Consider generating one pixel at a time, in sequence, through the image. My associative model tells me P(col of next pixel | all previous). Now, I can generate coffee cups images similar to any variation or combination of the images in the dataset. Now, you might say, "well you can only do that if you have a model of a coffee cup" (rather than of pixels) -- if so, just generate a coffee cup at one of the angles not in the dataset. This will not happen, because the model has not been provided with enough information to do so. Namely, the model does not know the distance from the camera, the camera lens parameters, the angle to the coffee cup, etc. So there's literally a very very large inifinity of possible objects at unseen angles. Consider that underneath a coffee cup, the bottom might be missing entirely, etc. Now it will appear to know all of these things, because its just generating images with these same parameters (camera, angle, distance, etc.). But as soon as you want "a coffee further away than has been seen before", or "a coffee using a macro lens", etc. the whole thing will fall over. It is you, the view, who attributes 3D knowledge to the model because under ordinary circumstances the cause of a photo is features of a 3D environment.
  - jameshart 2y ago
    You’re saying this with confidence as if there isn’t a large body of working image and video generation algorithms out there that can produce physically plausible images of objects transposed into circumstances that don’t exist in their training set. A coffee using a macro lens for example. Is it so hard to believe that such models have developed a sense for how light propagates through a scene, a sense for how physical objects change when viewed from different angles, a sense for how lens distortion interacts with light? For goodness’ sake, these same models have a sense of what Greg Rutkowski’s art style is - we are well beyond ‘they’re just remembering pixels from past coffeecups’
    - croon 2y ago
tiborsaas 2y ago
If your point is LLM-s are different than humans, I guess that goes without saying? We are not even on the same level yet. > the most common word in a corpus of text on which it was trained I think you are downplaying the fine grains of knowledge that can be encoded in a huge corpus of text. LLM-s are capable of taking context into account and encoding that too, not simply how often word A comes after B. When I'm in a conversation I'm also selecting the optimal word from a predefined dictionary. That's precisely what's like speaking in a given language. Sure, I'm thinking a bit ahead and I can tap into my memory, feelings and experiences which influences everything. But the optimal part is derived from context for me, it changes which word I use when I talk to a colleague, family or friend, but I might want to say the same thing. For stock LLM-s everything must be defined in the prompt if we are talking about zero-shot inference. These models are opening good insight on how language works and I don't find that too dehumanising. There's plenty of room exists still for me to be human and do non AI things. I get the notion that if we understand fully how something works the magic is gone, this always happens to AI. Are we afraid that this might happen to us too?
- mjburgess 2y ago
  > Are we afraid that this might happen to us too? No, I'm more frustrated by the pseudoscience that models of frequency associations in text are explanations of people (or anything else). The choice isn't between a pseudoscientific behaviouralism where animals have no presence in the world, no mental faculties, and so on vs. "magic". > When I'm in a conversation I'm also selecting the optimal word from a predefined dictionary Consider it this way: the probability dististribution over all possible world for you speaking is parameterized on space and time: Pyou(x, t; ...). And of the LLM generating text, Pllm(historical data). So imagine plotting the live probability distributions of Pyou and Pllm for any given situation. As you think, imagine, move, recall, prefer, desire... the Pyou goes "wild" with dramatic discontinous shifts in distribution brought about by these causes. Whereas the Pllm remains the same. It never changes. It never reacts to anything at all. The whole distribution over all prior text tokens, Pllm is a stationary model of frequency associations. Yours is not. This makes all the difference in the world when claiming that Pllm somehow models, or is even relevant to, Pyou.
throw310822 2y ago
> When the LLM generates each word it does so because it is, on average, the most common word in a corpus of text on which it was trained: "wearing" follows, "I like what you were" because most people who were having these conversations, captured in the training data, were talking about clothes. Yes. Now extend this to the context length of GPT-4 turbo- about 240 pages of text. So from your description, "wearing" is just the "most common word" to follow those 240 pages of (previously unseen, unique) text according to its training data. Quite simple, nothing to see here, I suppose.
chx 2y ago
https://hachyderm.io/@inthehands/112006855076082650 https://hachyderm.io/@inthehands/112006855076082650 > You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to. > Alas, that does not remotely resemble how people are pitching this technology.
gizmo 2y ago
Humans are capable of intricate complex thought that involves "mental states of recollection, preference, taste". However, LLMs have demonstrated pretty conclusively that most of the thoughts we have do not require any of that. Language turns out to be much simpler than previously assumed. Thinking hard it makes your brain hurt. It's exhausting. Most of the work we do, including programming, is not like that. Some of the work we do is fiendishly difficult but much of it is more like word-completion based on prior experience. Evolutionary processes optimize for energy efficiency. We don't think at 100% brain power all the time because we can't afford to. It makes a ton of sense, in retrospect, that our brains have optimized for language to the point that very little compute is required. And even so, the brain still consumes 20% of our daily calories. Hard thinking is the exception and casual thinking is the norm. Why is it so hard to persuade people of anything on the internet? Because we mostly engage in LLM-like auto-completion. Very little actual thinking is involved and very few calories are spent.
logicallee 2y ago
>When the LLM generates each word it does so because it is, on average, the most common word in a corpus of text on which it was trained: But ChatGPT 4 follows instructions passably well. For example I just asked it: "Construct a sentence of at least 10 words each of which is extremely grammatically unlikely to follow the word before it. (For example "be are isn't had" as each of those words is impossible after the word before it.) Do not give any explanation of how you have arrived at your answer, reply only with your answer. However, as you construct it ensure that you cannot think of any context in which each next word would ever come after the word before it. Reply with your constructed nonsense sentence only." Indeed it replied with a good nonsense sentence: "Dogs swimming beautifully reads soft under Wednesday during sky oranges" ("sky oranges" is unlikely, "under Wednesday" is nonsensical and ungrammatical) and when I complained that "dogs swimming" could be sensible as can "swimming beautifully" it came up with an even more nonsense sentence "Apples slowly would butter river quickly seven whenever blue music". Do you think "Wednesday" is really the most likely word to follow "under" and "river" is really the most likely to follow "butter", or isn't it obvious that it was, for lack of a better word, "trying to" follow my prompt? https://chat.openai.com/share/50037af6-0f3e-4de3-aff7-53a7b9a31c56 https://chat.openai.com/share/50037af6-0f3e-4de3-aff7-53a7b9...
- mjburgess 2y ago
  Well ChatGPT is a mixture of LLMs, and no doubt a great deal more to make this work out (i'd suppose, eg., that they augment their datasets to have conversational framing, with actions/verbs etc. augumented -- or they acheive similar with models-on-top). Nevertheless, roughly consider a dataset D for which we have an approximate stochastic model of its conditional frequency associations: P(next|previous..., D) etc. Then if your prompt really got that reply, from this model, it would do so like this: "Construct" is first projected to an encoding which replaces it, effectively, with a set of related words (Construct, Make, Create, Write...) all weighted by how they co-occur with construct. Then we sample from D based on this word set, obtaining roughly, all conversations where these related words were used, call this Dc. Next take "a sentence" and replace it with its word-set, say, (Sentence, Phrase, Words, ...) and sample conversations from Dc in which these occur, Dcs.. And so on. Since each token in your prompt actually corresponds to basically all possible words but weighted by association, each "filtering operation" actually selects vast amounts of the training data (space). Finally, consider the reverse problem: what words could this system possibly produce from this process that weren't relevant to your prompt? Given enough data (PBs of text from all possible digitized conversations, books, etc.) then a sensible-seeming answer becomes the only plausible one to generate. Now, I do think here PBs wouldnt be enough to generate a single statistical model that behaved this way -- so you need a mixture of them (ie., ChatGPT) and I suspect you also need a system for regulating discrete constraints such as quantities. I suspect many deployed LLMs have improved in this area due to models trained to be specifically sensitive to quantities.
eggdaft 2y ago
I think what this argument is missing is the emergent properties of the LLM. In order to “predict the next word”, the LLM doesn’t just learn the most likely word from a corpus for the preceding string. If that were true, it would not generalise outside of its training set. The LLM learns about the structure of the language, the context, and in the process of doing so constructs a model of the world as represented by words. Admittedly the model is still limited, but it seems to me that there is something more insightful to be gleaned here: that given enough data, and sufficient pressure to learn, that excelling at scale on a relatively simple task leads indirectly to a form of intelligence. For me the biggest takeaway of LLMs might be that “intelligence is pretty cheap, actually” and that the human brain is not so remarkable as we’d like to believe.
- euroderf 2y ago
  > and that the human brain is not so remarkable as we’d like to believe. Well, it IS pretty seamlessly integrated with a very impressive suite of sensors.
  - danielbln 2y ago
    Yes, our human sensor fusion is remarkable. The input signal of say our eyes is warped, upside down and low resolution apart from a tiny patch that races across the field of vision to capture high resolution samples (saccades). Yet, to us, it feels seamless and encompassing.
- ryandvm 2y ago
  Bingo. When I write some 100% bespoke code that is rather hastily composed and then paste it all into ChatGPT4 asking it to "refactor this code with a focus on testability and maintainability" and not only does it do so, but it does a pretty damn good job about it, it feels rather reductive to say "it's just providing the next most likely word". I mean, maybe that's how it works, but that statistical output clearly involves modeling what my code does and what I want it to do. Rather than make me think LLMs are a cheap trick, it just has me thinking, "shit - maybe that's all I do too."
ThinkingAgain 2y ago
Can it also explain the following: "An alien named abcdpqrs landed on earth yesterday. What is the name of the alien who landed yesterday on earth?" Chatgpt answers it correctly. abcdpqrs is perhaps not in the training set. If it is we can pick some othername.

jameshart 2y ago

LLMs don’t ’replay variations on prior conversations’ though. Predicting the probable token in a conversation requires predicting the probable subject of the conversation, predicting the interlocutors’ relationship and manner of speaking to one another, predicting the state of recollection, preference and taste of the speaker, predicting the speaker’s mental model… If the LLM isn’t predicting all of those things then it will produce poor predictions of the next word; doing it well - and humans tend to agree that in a vast array of cases state of the art LLMs do predict tokens very well - requires that prediction model to predict all that context as well.

mjburgess 2y ago
Alas, no it doesn't. Language induces this sort of anthropomorphism in people, I guess, so consider images. Suppose I take a billion images of all the coffee cups in the world, at a set of angles on the cup, and then build an associative (ie., frequency) statistical model of their pixels (ie., statistical AI). Consider generating one pixel at a time, in sequence, through the image. My associative model tells me P(col of next pixel | all previous). Now, I can generate coffee cups images similar to any variation or combination of the images in the dataset. Now, you might say, "well you can only do that if you have a model of a coffee cup" (rather than of pixels) -- if so, just generate a coffee cup at one of the angles not in the dataset. This will not happen, because the model has not been provided with enough information to do so. Namely, the model does not know the distance from the camera, the camera lens parameters, the angle to the coffee cup, etc. So there's literally a very very large inifinity of possible objects at unseen angles. Consider that underneath a coffee cup, the bottom might be missing entirely, etc. Now it will appear to know all of these things, because its just generating images with these same parameters (camera, angle, distance, etc.). But as soon as you want "a coffee further away than has been seen before", or "a coffee using a macro lens", etc. the whole thing will fall over. It is you, the view, who attributes 3D knowledge to the model because under ordinary circumstances the cause of a photo is features of a 3D environment.
- jameshart 2y ago
  You’re saying this with confidence as if there isn’t a large body of working image and video generation algorithms out there that can produce physically plausible images of objects transposed into circumstances that don’t exist in their training set. A coffee using a macro lens for example. Is it so hard to believe that such models have developed a sense for how light propagates through a scene, a sense for how physical objects change when viewed from different angles, a sense for how lens distortion interacts with light? For goodness’ sake, these same models have a sense of what Greg Rutkowski’s art style is - we are well beyond ‘they’re just remembering pixels from past coffeecups’
  - croon 2y ago

tiborsaas 2y ago

If your point is LLM-s are different than humans, I guess that goes without saying? We are not even on the same level yet. > the most common word in a corpus of text on which it was trained I think you are downplaying the fine grains of knowledge that can be encoded in a huge corpus of text. LLM-s are capable of taking context into account and encoding that too, not simply how often word A comes after B. When I'm in a conversation I'm also selecting the optimal word from a predefined dictionary. That's precisely what's like speaking in a given language. Sure, I'm thinking a bit ahead and I can tap into my memory, feelings and experiences which influences everything. But the optimal part is derived from context for me, it changes which word I use when I talk to a colleague, family or friend, but I might want to say the same thing. For stock LLM-s everything must be defined in the prompt if we are talking about zero-shot inference. These models are opening good insight on how language works and I don't find that too dehumanising. There's plenty of room exists still for me to be human and do non AI things. I get the notion that if we understand fully how something works the magic is gone, this always happens to AI. Are we afraid that this might happen to us too?

mjburgess 2y ago
> Are we afraid that this might happen to us too? No, I'm more frustrated by the pseudoscience that models of frequency associations in text are explanations of people (or anything else). The choice isn't between a pseudoscientific behaviouralism where animals have no presence in the world, no mental faculties, and so on vs. "magic". > When I'm in a conversation I'm also selecting the optimal word from a predefined dictionary Consider it this way: the probability dististribution over all possible world for you speaking is parameterized on space and time: Pyou(x, t; ...). And of the LLM generating text, Pllm(historical data). So imagine plotting the live probability distributions of Pyou and Pllm for any given situation. As you think, imagine, move, recall, prefer, desire... the Pyou goes "wild" with dramatic discontinous shifts in distribution brought about by these causes. Whereas the Pllm remains the same. It never changes. It never reacts to anything at all. The whole distribution over all prior text tokens, Pllm is a stationary model of frequency associations. Yours is not. This makes all the difference in the world when claiming that Pllm somehow models, or is even relevant to, Pyou.

throw310822 2y ago

> When the LLM generates each word it does so because it is, on average, the most common word in a corpus of text on which it was trained: "wearing" follows, "I like what you were" because most people who were having these conversations, captured in the training data, were talking about clothes. Yes. Now extend this to the context length of GPT-4 turbo- about 240 pages of text. So from your description, "wearing" is just the "most common word" to follow those 240 pages of (previously unseen, unique) text according to its training data. Quite simple, nothing to see here, I suppose.

chx 2y ago

https://hachyderm.io/@inthehands/112006855076082650 https://hachyderm.io/@inthehands/112006855076082650 > You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to. > Alas, that does not remotely resemble how people are pitching this technology.

gizmo 2y ago

Humans are capable of intricate complex thought that involves "mental states of recollection, preference, taste". However, LLMs have demonstrated pretty conclusively that most of the thoughts we have do not require any of that. Language turns out to be much simpler than previously assumed. Thinking hard it makes your brain hurt. It's exhausting. Most of the work we do, including programming, is not like that. Some of the work we do is fiendishly difficult but much of it is more like word-completion based on prior experience. Evolutionary processes optimize for energy efficiency. We don't think at 100% brain power all the time because we can't afford to. It makes a ton of sense, in retrospect, that our brains have optimized for language to the point that very little compute is required. And even so, the brain still consumes 20% of our daily calories. Hard thinking is the exception and casual thinking is the norm. Why is it so hard to persuade people of anything on the internet? Because we mostly engage in LLM-like auto-completion. Very little actual thinking is involved and very few calories are spent.

logicallee 2y ago

>When the LLM generates each word it does so because it is, on average, the most common word in a corpus of text on which it was trained: But ChatGPT 4 follows instructions passably well. For example I just asked it: "Construct a sentence of at least 10 words each of which is extremely grammatically unlikely to follow the word before it. (For example "be are isn't had" as each of those words is impossible after the word before it.) Do not give any explanation of how you have arrived at your answer, reply only with your answer. However, as you construct it ensure that you cannot think of any context in which each next word would ever come after the word before it. Reply with your constructed nonsense sentence only." Indeed it replied with a good nonsense sentence: "Dogs swimming beautifully reads soft under Wednesday during sky oranges" ("sky oranges" is unlikely, "under Wednesday" is nonsensical and ungrammatical) and when I complained that "dogs swimming" could be sensible as can "swimming beautifully" it came up with an even more nonsense sentence "Apples slowly would butter river quickly seven whenever blue music". Do you think "Wednesday" is really the most likely word to follow "under" and "river" is really the most likely to follow "butter", or isn't it obvious that it was, for lack of a better word, "trying to" follow my prompt? https://chat.openai.com/share/50037af6-0f3e-4de3-aff7-53a7b9a31c56 https://chat.openai.com/share/50037af6-0f3e-4de3-aff7-53a7b9...

mjburgess 2y ago
Well ChatGPT is a mixture of LLMs, and no doubt a great deal more to make this work out (i'd suppose, eg., that they augment their datasets to have conversational framing, with actions/verbs etc. augumented -- or they acheive similar with models-on-top). Nevertheless, roughly consider a dataset D for which we have an approximate stochastic model of its conditional frequency associations: P(next|previous..., D) etc. Then if your prompt really got that reply, from this model, it would do so like this: "Construct" is first projected to an encoding which replaces it, effectively, with a set of related words (Construct, Make, Create, Write...) all weighted by how they co-occur with construct. Then we sample from D based on this word set, obtaining roughly, all conversations where these related words were used, call this Dc. Next take "a sentence" and replace it with its word-set, say, (Sentence, Phrase, Words, ...) and sample conversations from Dc in which these occur, Dcs.. And so on. Since each token in your prompt actually corresponds to basically all possible words but weighted by association, each "filtering operation" actually selects vast amounts of the training data (space). Finally, consider the reverse problem: what words could this system possibly produce from this process that weren't relevant to your prompt? Given enough data (PBs of text from all possible digitized conversations, books, etc.) then a sensible-seeming answer becomes the only plausible one to generate. Now, I do think here PBs wouldnt be enough to generate a single statistical model that behaved this way -- so you need a mixture of them (ie., ChatGPT) and I suspect you also need a system for regulating discrete constraints such as quantities. I suspect many deployed LLMs have improved in this area due to models trained to be specifically sensitive to quantities.

eggdaft 2y ago

I think what this argument is missing is the emergent properties of the LLM. In order to “predict the next word”, the LLM doesn’t just learn the most likely word from a corpus for the preceding string. If that were true, it would not generalise outside of its training set. The LLM learns about the structure of the language, the context, and in the process of doing so constructs a model of the world as represented by words. Admittedly the model is still limited, but it seems to me that there is something more insightful to be gleaned here: that given enough data, and sufficient pressure to learn, that excelling at scale on a relatively simple task leads indirectly to a form of intelligence. For me the biggest takeaway of LLMs might be that “intelligence is pretty cheap, actually” and that the human brain is not so remarkable as we’d like to believe.

euroderf 2y ago
> and that the human brain is not so remarkable as we’d like to believe. Well, it IS pretty seamlessly integrated with a very impressive suite of sensors.
- danielbln 2y ago
  Yes, our human sensor fusion is remarkable. The input signal of say our eyes is warped, upside down and low resolution apart from a tiny patch that races across the field of vision to capture high resolution samples (saccades). Yet, to us, it feels seamless and encompassing.
ryandvm 2y ago
Bingo. When I write some 100% bespoke code that is rather hastily composed and then paste it all into ChatGPT4 asking it to "refactor this code with a focus on testability and maintainability" and not only does it do so, but it does a pretty damn good job about it, it feels rather reductive to say "it's just providing the next most likely word". I mean, maybe that's how it works, but that statistical output clearly involves modeling what my code does and what I want it to do. Rather than make me think LLMs are a cheap trick, it just has me thinking, "shit - maybe that's all I do too."

ThinkingAgain 2y ago

Can it also explain the following: "An alien named abcdpqrs landed on earth yesterday. What is the name of the alien who landed yesterday on earth?" Chatgpt answers it correctly. abcdpqrs is perhaps not in the training set. If it is we can pick some othername.