3 ms·

>When the LLM generates each word it does so because it is, on average, the most common word in a corpus of text on which it was trained: But ChatGPT 4 follows

>When the LLM generates each word it does so because it is, on average, the most common word in a corpus of text on which it was trained: But ChatGPT 4 follows instructions passably well. For example I just asked it: "Construct a sentence of at least 10 words each of which is extremely grammatically unlikely to follow the word before it. (For example "be are isn't had" as each of those words is impossible after the word before it.) Do not give any explanation of how you have arrived at your answer, reply only with your answer. However, as you construct it ensure that you cannot think of any context in which each next word would ever come after the word before it. Reply with your constructed nonsense sentence only." Indeed it replied with a good nonsense sentence: "Dogs swimming beautifully reads soft under Wednesday during sky oranges" ("sky oranges" is unlikely, "under Wednesday" is nonsensical and ungrammatical) and when I complained that "dogs swimming" could be sensible as can "swimming beautifully" it came up with an even more nonsense sentence "Apples slowly would butter river quickly seven whenever blue music". Do you think "Wednesday" is really the most likely word to follow "under" and "river" is really the most likely to follow "butter", or isn't it obvious that it was, for lack of a better word, "trying to" follow my prompt? https://chat.openai.com/share/50037af6-0f3e-4de3-aff7-53a7b9a31c56 https://chat.openai.com/share/50037af6-0f3e-4de3-aff7-53a7b9...

mjburgess 2y ago
Well ChatGPT is a mixture of LLMs, and no doubt a great deal more to make this work out (i'd suppose, eg., that they augment their datasets to have conversational framing, with actions/verbs etc. augumented -- or they acheive similar with models-on-top). Nevertheless, roughly consider a dataset D for which we have an approximate stochastic model of its conditional frequency associations: P(next|previous..., D) etc. Then if your prompt really got that reply, from this model, it would do so like this: "Construct" is first projected to an encoding which replaces it, effectively, with a set of related words (Construct, Make, Create, Write...) all weighted by how they co-occur with construct. Then we sample from D based on this word set, obtaining roughly, all conversations where these related words were used, call this Dc. Next take "a sentence" and replace it with its word-set, say, (Sentence, Phrase, Words, ...) and sample conversations from Dc in which these occur, Dcs.. And so on. Since each token in your prompt actually corresponds to basically all possible words but weighted by association, each "filtering operation" actually selects vast amounts of the training data (space). Finally, consider the reverse problem: what words could this system possibly produce from this process that weren't relevant to your prompt? Given enough data (PBs of text from all possible digitized conversations, books, etc.) then a sensible-seeming answer becomes the only plausible one to generate. Now, I do think here PBs wouldnt be enough to generate a single statistical model that behaved this way -- so you need a mixture of them (ie., ChatGPT) and I suspect you also need a system for regulating discrete constraints such as quantities. I suspect many deployed LLMs have improved in this area due to models trained to be specifically sensitive to quantities.
- danielbln 2y ago
  There are plenty of LLMs that aren't MoE/ensemble, and there are also plenty of LLMs that are pure completion models, that haven't been fine-tuned/RLHF'd to be conversational. I would recommend you read up a bit more on how modern LLMs work, I get the feeling your intuition on that could improve. edit: I can't reply to the child comment as we've reached the thread limit, but I can say that LLMs are not trained on a tiny subset of data, they are trained on as much data as possible. A LLM becomes converational/instruct due to fine tuning it with reinforcement learning data. GPT-3.5 is by all accounts not an ensemble model, Llama2/3 is NOT an ensemble model/MoE, yet will allow you to do in-context learning/few shot prompting effortlessly. As said, I think your intuition on how these LLMs work and (as far as we know) how they work, needs readjustment.
  - mjburgess 2y ago
    I dont see what I'm missing. I'm addressing why ChatGPT generated a response given a prompt. If another LLM had been used, something far simpler, the explanation would be different. If a highly simplified LLM will generate text against discrete quantitative constraints, under a variety of scenarios, then I've underestimated how highly structured the relevant training data must be. An LLM trained on a physics textbook isnt going to be conversational; one trained on shakespear will generate text from elizabethan english.. ie., in every case, the explanation of why any given response was generated is given by explaining the distribution of its dataset. So if a shakespear LLM generates, "to be or otherwise to be not is alike everything ere annon" we will be mostly explaining how/why those words were used by shakespear. and if an LLM is small, and is actually discretely sensitive to quantities across a large vareity of domains.. my guess is that its training data has been specially prepared. This is jsut a guess about hte nature of human commnuication though, it has nothing to do with LLMs. I just guess that we don't distribute "quantity tokens" in such a highly patterned way that a simple LLM model would work to find it
danielbln 2y ago
Yeah, I feel OP is ignoring the magic of in-context learning in their slightly reductive view on how LLMs work.
- adammarples 2y ago
  They're basically describing a markov chain with word tokenisation. Which is so remarkably out of date compared to how a modern got works.
- mjburgess 2y ago
  The purpose of my original comment wasn't to accurately depict LLMs, but to introduce the properties of people that cause us to write/speak etc. which LLMs aren't sensitive to. The point was to answer the question, "in what ways arent we just doing the same?" The point of the LLM bit is that the property of the world that LLMs are sensitive to is the distribution of text tokens in their training data. Regardless of which features of this distribution any given AI model captures, it is necessarily a model of the dataset's actual P(token|tokens..). In the case of LLMs its a very high dimensional model so that P(word|previous words) is actually modelled by something like: P(word|P(prompt embedding space|answer embedding space) | ...) -- but this makes no difference to the "arent we doing the same?" question. We dont use frequency associations between parts of a historical corpus when we speak.
  - TeMPOraL 2y ago
    Everything you say here makes sense, except the last bit: > We dont use frequency associations between parts of a historical corpus when we speak. But that's the thing, it seems we do. Arguably, the very meaning of concepts is determined solely by associations with other concepts, in a way remarkably similar if not identical to frequency associations.
    - mjburgess 2y ago
      No, no.. the semantics of words is not other words. Cavemen wander around, they fall over a pig, they point to pig and say "pig". Other cavemen observe. Later, when they want a pig, they say "pig". No one here knows anything about pigs other than that there is something in the world which causes people to say "pig" and each caveman is able to locate that thing after awhile. The vast majority of language is nothing more than this: words point outside themselves to the world, this pointing is grown in us through acquaintance with the world. Now, in general, the cause of my saying "pig" is not me falling over one. Suppose I say, to a friend, "I've always thought pigs were cute, until i saw a big one!" So here, "I" points at both me as a body, but also plausibly at my model of myself (etc.), "always" modifies "thought" ... so "I've always thought" ends up being a statement about how my own models of my self over time have changed.. and so on for "pigs" and the like. We do not know that this is what our words mean. We have no idea that what we're referring to when I say "I've always thought" -- the nature of the world that our words refers to requires, in general, science to explain. Words are, at first, just a familiar way of throwing darts at a target which we can see, but not describe nor explain. It is this process which is entirely absent in an LLM. An LLM isnt throwing a dart at anything, it isnt even speaking. It's replaying historical darts matches between people. And this is just to consider reference. There are other causes of our using words much more complex than our trying to refer to things, likewise, these are absent from the LLM.

mjburgess 2y ago

Well ChatGPT is a mixture of LLMs, and no doubt a great deal more to make this work out (i'd suppose, eg., that they augment their datasets to have conversational framing, with actions/verbs etc. augumented -- or they acheive similar with models-on-top). Nevertheless, roughly consider a dataset D for which we have an approximate stochastic model of its conditional frequency associations: P(next|previous..., D) etc. Then if your prompt really got that reply, from this model, it would do so like this: "Construct" is first projected to an encoding which replaces it, effectively, with a set of related words (Construct, Make, Create, Write...) all weighted by how they co-occur with construct. Then we sample from D based on this word set, obtaining roughly, all conversations where these related words were used, call this Dc. Next take "a sentence" and replace it with its word-set, say, (Sentence, Phrase, Words, ...) and sample conversations from Dc in which these occur, Dcs.. And so on. Since each token in your prompt actually corresponds to basically all possible words but weighted by association, each "filtering operation" actually selects vast amounts of the training data (space). Finally, consider the reverse problem: what words could this system possibly produce from this process that weren't relevant to your prompt? Given enough data (PBs of text from all possible digitized conversations, books, etc.) then a sensible-seeming answer becomes the only plausible one to generate. Now, I do think here PBs wouldnt be enough to generate a single statistical model that behaved this way -- so you need a mixture of them (ie., ChatGPT) and I suspect you also need a system for regulating discrete constraints such as quantities. I suspect many deployed LLMs have improved in this area due to models trained to be specifically sensitive to quantities.

danielbln 2y ago
There are plenty of LLMs that aren't MoE/ensemble, and there are also plenty of LLMs that are pure completion models, that haven't been fine-tuned/RLHF'd to be conversational. I would recommend you read up a bit more on how modern LLMs work, I get the feeling your intuition on that could improve. edit: I can't reply to the child comment as we've reached the thread limit, but I can say that LLMs are not trained on a tiny subset of data, they are trained on as much data as possible. A LLM becomes converational/instruct due to fine tuning it with reinforcement learning data. GPT-3.5 is by all accounts not an ensemble model, Llama2/3 is NOT an ensemble model/MoE, yet will allow you to do in-context learning/few shot prompting effortlessly. As said, I think your intuition on how these LLMs work and (as far as we know) how they work, needs readjustment.
- mjburgess 2y ago
  I dont see what I'm missing. I'm addressing why ChatGPT generated a response given a prompt. If another LLM had been used, something far simpler, the explanation would be different. If a highly simplified LLM will generate text against discrete quantitative constraints, under a variety of scenarios, then I've underestimated how highly structured the relevant training data must be. An LLM trained on a physics textbook isnt going to be conversational; one trained on shakespear will generate text from elizabethan english.. ie., in every case, the explanation of why any given response was generated is given by explaining the distribution of its dataset. So if a shakespear LLM generates, "to be or otherwise to be not is alike everything ere annon" we will be mostly explaining how/why those words were used by shakespear. and if an LLM is small, and is actually discretely sensitive to quantities across a large vareity of domains.. my guess is that its training data has been specially prepared. This is jsut a guess about hte nature of human commnuication though, it has nothing to do with LLMs. I just guess that we don't distribute "quantity tokens" in such a highly patterned way that a simple LLM model would work to find it

danielbln 2y ago

Yeah, I feel OP is ignoring the magic of in-context learning in their slightly reductive view on how LLMs work.

adammarples 2y ago
They're basically describing a markov chain with word tokenisation. Which is so remarkably out of date compared to how a modern got works.
mjburgess 2y ago
The purpose of my original comment wasn't to accurately depict LLMs, but to introduce the properties of people that cause us to write/speak etc. which LLMs aren't sensitive to. The point was to answer the question, "in what ways arent we just doing the same?" The point of the LLM bit is that the property of the world that LLMs are sensitive to is the distribution of text tokens in their training data. Regardless of which features of this distribution any given AI model captures, it is necessarily a model of the dataset's actual P(token|tokens..). In the case of LLMs its a very high dimensional model so that P(word|previous words) is actually modelled by something like: P(word|P(prompt embedding space|answer embedding space) | ...) -- but this makes no difference to the "arent we doing the same?" question. We dont use frequency associations between parts of a historical corpus when we speak.
- TeMPOraL 2y ago
  Everything you say here makes sense, except the last bit: > We dont use frequency associations between parts of a historical corpus when we speak. But that's the thing, it seems we do. Arguably, the very meaning of concepts is determined solely by associations with other concepts, in a way remarkably similar if not identical to frequency associations.
  - mjburgess 2y ago
    No, no.. the semantics of words is not other words. Cavemen wander around, they fall over a pig, they point to pig and say "pig". Other cavemen observe. Later, when they want a pig, they say "pig". No one here knows anything about pigs other than that there is something in the world which causes people to say "pig" and each caveman is able to locate that thing after awhile. The vast majority of language is nothing more than this: words point outside themselves to the world, this pointing is grown in us through acquaintance with the world. Now, in general, the cause of my saying "pig" is not me falling over one. Suppose I say, to a friend, "I've always thought pigs were cute, until i saw a big one!" So here, "I" points at both me as a body, but also plausibly at my model of myself (etc.), "always" modifies "thought" ... so "I've always thought" ends up being a statement about how my own models of my self over time have changed.. and so on for "pigs" and the like. We do not know that this is what our words mean. We have no idea that what we're referring to when I say "I've always thought" -- the nature of the world that our words refers to requires, in general, science to explain. Words are, at first, just a familiar way of throwing darts at a target which we can see, but not describe nor explain. It is this process which is entirely absent in an LLM. An LLM isnt throwing a dart at anything, it isnt even speaking. It's replaying historical darts matches between people. And this is just to consider reference. There are other causes of our using words much more complex than our trying to refer to things, likewise, these are absent from the LLM.