4 ms·

“ When an LLM is suggesting what might come next in a piece of text... it doesn't know if it's supposed to guess a probable word from a Wikipedia article, an On

“ When an LLM is suggesting what might come next in a piece of text... it doesn't know if it's supposed to guess a probable word from a Wikipedia article, an Onion article, a Project Gutenberg manuscript, or an Archive Of Our Own fanfic” LLMs are very good at inferring context, so that only really applies if you’re using an un-RLHFed base model with no context given

mjburgess 3y ago
Here, "supposed to guess" means "having the goal of..." So no LLM knows what it's supposed to do. If you prefer, you could say it only ever has one goal: to generate a sequence of tokens which are jointly the most probable to occur along with the prompt tokens, given such probabilities in a historical corpus. This imitates knowledge, goal-directness, "inferring context" etc. without doing any of those things. Consider what the aim of knowing, goal-directness, inferring , etc. is --- it is never "consistency with a historical text corpus". For knowing: that beliefs correspond to the way the world is; for goal-directness that one's acts+desires can realise changes; for 'inferring context': that one is sensitive to reasons to speak outside of what is literally spoken. LLMs are never sensitive to reasons to speak outside of what has been spoken.
- Ajedi32 3y ago
  What does RLHF do then? I feel like you completely ignored the central point of GP's comment. RLHF is the difference between GPT-3.5 and ChatGPT, and it's the whole reason why LLMs are suddenly such a big deal. ChatGPT demonstrated that it's possible to give language models a goal beyond just "complete most likely next word" and that they can actually be somewhat competent at achieving those goals despite not being explicitly trained for them.
  - mjburgess 3y ago
    > competent at achieving those goals despite not being explicitly trained for them. Well (1) it doesn't achieve goals, since a "goal" is observer-relative. We have goals, the LLM has a formal optimisation objective which gives it the appearence of goal-directed behaviour (in a similar way, eg., that it appears pens want to fall when dropped). And (2), reading your "goal" here even in observer-relative ways, I don't think there's much evidence of this. These models are "trained" on everything ever written, include all of the internet and basically all digitised book. I don't see any evidence of much generalisation -- if you can find it by google, then the LLM has it stored compressed (ie., the "weights"). The innovation in LLMs is being able to compute `max P(answer|prompt, historical_corpus)` for increasingly longer prompts --- there's no innovation in goal-directed behaviour. That's VC propangada to disguise the fact that LLMs are mostly an innovation in copyright laundering.
    - Ajedi32 3y ago
      (1) This is a tired, pointless semantic argument. "It doesn't have a goal, it just acts like it has a goal for all intents and purposes. But, you see, it's actually a machine and not a human and therefore it can't really have goals according to my narrow definition of the term." Either point to an actually relevant difference in the resulting behavior or stop objecting when people use human behavioral terms to describe the behavior of machine learning systems. We're all well aware it's a program; that's not the point. (Sorry, just a frustration I have with the larger discussion around this topic.) (2) "I don't see any evidence of much generalisation" Seriously? So when I tell ChatGPT to rewrite a paragraph in the style of Shakespeare and it does it, despite never being trained to do that, never seeing the source or target paragraph before, and having no information other than my text prompt and its past training, that's not evidence of generalization? And that's only one of millions of different possible tasks that the same model excels at, despite being trained on nothing but a bunch of unstructured text and a few examples indicating its goal should be to follow instructions given in the prompt text. Up until a couple years ago this level of flexibility in a machine learning model would have been considered science fiction by nearly everyone, and now it's "[not] evidence of much generalization". Okay.
      - mjburgess 3y ago
        Well (1), the reason this distinction is relevant is so we can separate out whether the system has developed a capacity or an apparent capacity. Is the child a genius or are they just reading out of a textbook? Can the toddler really compose a sonata or did they just press play on the piano keyboard? (2) This is indeed the power of interpolating between the data points of "everything ever written in human history" as digitised and compressed by ChatGPT. If you have 1 billion circles of radii 0 to 1, it isn't generalisation for the machine to produce one with a radii 0.0000100003000001, ie., one not in the set but a mere interpolation of points within it. It would be expensive, but imagining "reversing" ChatGPT from it's output to the sources which made a non-trivial difference to generating that output. So the function there is: response -> verbatim text in the training corpus. Then, maybe, "bolded" by how much each paragraph would "make a difference" to its output. What you'd find is thousands of pages: all Shakespeare ever written, all papers about Shakespeare, all books about Shakespeare; and so on. Then when it applied the bolding, and summarising it a little, the trick would be revealed: it would be apparent how a naive statistical interpolation between sequences of characters could produce the effect. ChatGPT exists because of ebooks and social media: without it, it could do almost nothing. That is, the appearance of these capacities is strictly derivative of the work of a billion people who had them. Without vast, unimaginable, amounts of work produced on Shakespeare this system wouldnt work. It's just a copyright laundering system. All the school essays on reddit, all the forum posts; all of usenet. All pdfs, all digitised works. All academic papers. Is this generalisation? Is this a system which starts with little and makes a lot? Or is it a system which is more like a child reading from a textbook? Ie., making a haphazard ability to repeat what's already written. The size of the weights of a modern LLM are sufficient to compress everything ever written in human history: and that's exactly what they do.

mjburgess 3y ago

Here, "supposed to guess" means "having the goal of..." So no LLM knows what it's supposed to do. If you prefer, you could say it only ever has one goal: to generate a sequence of tokens which are jointly the most probable to occur along with the prompt tokens, given such probabilities in a historical corpus. This imitates knowledge, goal-directness, "inferring context" etc. without doing any of those things. Consider what the aim of knowing, goal-directness, inferring , etc. is --- it is never "consistency with a historical text corpus". For knowing: that beliefs correspond to the way the world is; for goal-directness that one's acts+desires can realise changes; for 'inferring context': that one is sensitive to reasons to speak outside of what is literally spoken. LLMs are never sensitive to reasons to speak outside of what has been spoken.

Ajedi32 3y ago
What does RLHF do then? I feel like you completely ignored the central point of GP's comment. RLHF is the difference between GPT-3.5 and ChatGPT, and it's the whole reason why LLMs are suddenly such a big deal. ChatGPT demonstrated that it's possible to give language models a goal beyond just "complete most likely next word" and that they can actually be somewhat competent at achieving those goals despite not being explicitly trained for them.
- mjburgess 3y ago
  > competent at achieving those goals despite not being explicitly trained for them. Well (1) it doesn't achieve goals, since a "goal" is observer-relative. We have goals, the LLM has a formal optimisation objective which gives it the appearence of goal-directed behaviour (in a similar way, eg., that it appears pens want to fall when dropped). And (2), reading your "goal" here even in observer-relative ways, I don't think there's much evidence of this. These models are "trained" on everything ever written, include all of the internet and basically all digitised book. I don't see any evidence of much generalisation -- if you can find it by google, then the LLM has it stored compressed (ie., the "weights"). The innovation in LLMs is being able to compute `max P(answer|prompt, historical_corpus)` for increasingly longer prompts --- there's no innovation in goal-directed behaviour. That's VC propangada to disguise the fact that LLMs are mostly an innovation in copyright laundering.
  - Ajedi32 3y ago
    (1) This is a tired, pointless semantic argument. "It doesn't have a goal, it just acts like it has a goal for all intents and purposes. But, you see, it's actually a machine and not a human and therefore it can't really have goals according to my narrow definition of the term." Either point to an actually relevant difference in the resulting behavior or stop objecting when people use human behavioral terms to describe the behavior of machine learning systems. We're all well aware it's a program; that's not the point. (Sorry, just a frustration I have with the larger discussion around this topic.) (2) "I don't see any evidence of much generalisation" Seriously? So when I tell ChatGPT to rewrite a paragraph in the style of Shakespeare and it does it, despite never being trained to do that, never seeing the source or target paragraph before, and having no information other than my text prompt and its past training, that's not evidence of generalization? And that's only one of millions of different possible tasks that the same model excels at, despite being trained on nothing but a bunch of unstructured text and a few examples indicating its goal should be to follow instructions given in the prompt text. Up until a couple years ago this level of flexibility in a machine learning model would have been considered science fiction by nearly everyone, and now it's "[not] evidence of much generalization". Okay.
    - mjburgess 3y ago
      Well (1), the reason this distinction is relevant is so we can separate out whether the system has developed a capacity or an apparent capacity. Is the child a genius or are they just reading out of a textbook? Can the toddler really compose a sonata or did they just press play on the piano keyboard? (2) This is indeed the power of interpolating between the data points of "everything ever written in human history" as digitised and compressed by ChatGPT. If you have 1 billion circles of radii 0 to 1, it isn't generalisation for the machine to produce one with a radii 0.0000100003000001, ie., one not in the set but a mere interpolation of points within it. It would be expensive, but imagining "reversing" ChatGPT from it's output to the sources which made a non-trivial difference to generating that output. So the function there is: response -> verbatim text in the training corpus. Then, maybe, "bolded" by how much each paragraph would "make a difference" to its output. What you'd find is thousands of pages: all Shakespeare ever written, all papers about Shakespeare, all books about Shakespeare; and so on. Then when it applied the bolding, and summarising it a little, the trick would be revealed: it would be apparent how a naive statistical interpolation between sequences of characters could produce the effect. ChatGPT exists because of ebooks and social media: without it, it could do almost nothing. That is, the appearance of these capacities is strictly derivative of the work of a billion people who had them. Without vast, unimaginable, amounts of work produced on Shakespeare this system wouldnt work. It's just a copyright laundering system. All the school essays on reddit, all the forum posts; all of usenet. All pdfs, all digitised works. All academic papers. Is this generalisation? Is this a system which starts with little and makes a lot? Or is it a system which is more like a child reading from a textbook? Ie., making a haphazard ability to repeat what's already written. The size of the weights of a modern LLM are sufficient to compress everything ever written in human history: and that's exactly what they do.