4 ms·

> LLMs are booksmart Harvard graduates who can Google anything I wouldn't say this. Internet savvy Harvard graduates have common sense. They can look at someth

> LLMs are booksmart Harvard graduates who can Google anything I wouldn't say this. Internet savvy Harvard graduates have common sense. They can look at something and mostly infer "wait, that's not right," and error correct, or admit low confidence in the answer. I like to think of LLMs as hardcore improv actors. They have a script (the context). Their burning desire is to continue that script. And they will just roll with the best continuation they can, whatever that continuation is. OpenAI's augmentations hand them a very dynamic script, but at the end of the day they could just be madly improvising for all you know.

haswell 3y ago
> but at the end of the day they could just be madly improvising for all you know I think it's important to draw a line between the kind of madness and improvisation a human performs in these situations vs. an LLM. While it's still very possible that the LLM spits out something entirely wrong, the reasons for this wrongness are going to be quite different than the reasons a human performing improv will improvise something - anything - that keeps the improvisation going. The human is still fundamentally limited by the corpus of knowledge available to them, and by the limits of human cognition in processing all available information and generating a result. While I think the general point of thinking of an LLM as improvising vs. Googling is interesting to consider, I don't think it gets us closer to a useful mental model. A good LLM is still going to be right about most of the things it "improvises", even though it doesn't "know" anything. The person doing improv is not concerned about saying things that are factually true, and is focused on dynamically generating a story or joke or narrative based only on the immediate context. So while I do recognize some similarities, I don't think the similarities help us think about LLMs in a more useful way.
- peteradio 3y ago
  It sounds like you agree then and its only a matter of the corpus at hand. An improv actor would be called a comedian if they tried to interpolate based on spotty context.
  - haswell 3y ago
    It’s a matter of the corpus at hand and the predictability of a human vs. a computer. With more training and improved algorithms, the computer can reach a high rate of accurate “improvisation”, at which point it doesn’t matter if it’s improv or not. My issue is with using improv to augment our mental models of LLMs, and I don’t agree that this is the end result. Taking this analogy too far misleads us about LLMs.
- brucethemoose2 3y ago
  > The person doing improv is not concerned about saying things that are factually true, and is focused on dynamically generating a story or joke or narrative based only on the immediate context. I disagree, as this is exactly what LLMs do! Its literally the function of a text completion transformers model. OpenAI has just "trained" their improv actor to speak like an an assistant and educated it well. They tell the model to roleplay an assistant in the script, and injects all sorts of facts and formatting into their script the user doesn't see. I believe they constrain grammar and post process it too. But ultimately the text is written by a heavily constrained improv actor that will absolutely improvise anything to keep the text flowing until the stop token arrives, even if truth from memory/context is usually the most likely improvisation.
  - haswell 3y ago
    Except this is not exactly what LLMs do. This is quickly crossing into an anthropomorphism fallacy. The thing that matters most is how accurate (or not) the LLM is, regardless of how it arrived at its output. A human doing improv bears some surface level resemblance to an LLM processing text, but neither the context nor the mechanics are actually similar, nor are the theoretical limits of computing power. The broader point is that this mental model doesn’t help us better understand LLMs, and as LLMs continue to improve, any resemblance to “improv” will be less and less relevant (not to mention inaccurate).
    - brucethemoose2 3y ago
      I don't mean to anthropomorphize it, but I am trying to drive the point that current LLMs are text completion models. The current architectures continue a script, one token at a time... that's what it does, even if its a trillion trillion parameters and heavily aligned on all all accurate human communication ever. And that's similar enough to the basic mental loop of a "trained" improv actor to serve as a metaphor, I think.
      - haswell 3y ago
        > And that's similar enough to the basic mental loop of a "trained" improv actor My point is that this is only superficially similar, and to claim deeper similarity is the anthropomorphic fallacy. To form a useful mental model, the mechanics need to be similar enough to inform the person evaluating the system as the underlying system evolves. As LLMs evolve, the improv analogy becomes less and less useful because the LLM gets more and more accurate while the person doing improv is still just doing improv. The major difference being that people doing improv aren’t trying to be oracles of information. Perfecting the craft of improv may have nothing to do with accuracy, only believability or humor. More generally, thinking of LLMs as analogous to intelligent humans introduces other misconceptions, leads people to over-estimate accuracy, etc. The oddity at the bottom of all of this is that eventually, LLMs may be considered accurate enough to be used reliably as sources of information despite this “improvisation”, at which point it’s not really fair to call it improv, which would massively undersell the utility of the tool. But a perfectly accurate LLM may still not match any of the primary characteristics of an intelligent human, and it won’t matter as long as it’s useful.
- recursive 3y ago
  > A good LLM is still going to be right about most of the things it "improvises", even though it doesn't "know" anything. When I check on the factual details of what ChatGPT tells me, it seems to be wrong a lot of the time. Maybe not most, but I've stopped relying on it for factual information.
  - haswell 3y ago
    I think this is a matter of framing. When compared to a system that does nothing but assembles grammatically correct sentences, the correctness of LLM output is impressively high. But ultimately I’m not saying that the accuracy of LLMs is sufficient, just that thinking about LLMs as improv machines isn’t very helpful.
    - seba_dos1 3y ago
      It has an encyclopedia around and a bit of human-scale time to consult it before coming up with the next line, but it's still more of an improv actor playing an expert than an actual expert. Sufficiently clever improv actor with sufficiently complete encyclopedia will at some point become indistinguishable from actual experts, but LLMs are still far from that point in general. They can already be great at fooling non-experts though, like an actor that needs to be just factually-correct enough to be believable in their role.
baq 3y ago
(In Soviet Russia, improv actors inject prompts into you.) LLMs are gullible, naive and will do anything to please you, because that's how they are trained. They're dogs who speak very good English and memoized the Internet.
- blitz_skull 3y ago
  I keep hearing these comparisons, and maybe on paper it’s “true” but if that’s the case; a significant portion of our actual problems can be exported to English dogs with sufficient memory. I regularly have hard problems that I can digest into chunks that LLMs routinely help me categorize, solve, and document. I feel like these comparisons are just not in touch with the actual amount of help that LLMs provide in a day to day context with programming related tasks.
  - jdashg 3y ago
    It's true, most tasks are just simpler chunks of reasoning that anyone can do, tied together with key insights that are harder to train.
  - efnx 3y ago
    Can you give an example because it has been my experience that they tend to give answers that sound right, but after closer inspection (or attempted compilation - or running) are in fact wrong.
    - baq 3y ago
      Dogs are obviously (I hope) a joke, actually they’re word calculators. They have some knowledge embedded because that’s needed to be able to be a word calculator but getting it facts in the context window works much better, hence they’re good at summarizing etc.
    - LeifCarrotson 3y ago
      They occasionally give answers that sound right but are in fact wrong. Their suitability for a particular task depends largely on your tolerance for that category of error. If you're generating fiction, it's irrelevant, if you're generating a newsletter that might be skimmed at a high level by disinterested people (or other LLMs) it's a low risk, if you're writing UI code it can be bad but will probably be corrected if it's bad enough, if you're writing critical code an insidious error can be disastrous.
all2 3y ago
My mental model of LLMs presently is as a "language core". You wouldn't trust your own stream of consciousness and you have checks and balances in your brain that you use to reason. In the same manner I don't really trust an LLM to "reason" unless there's a software/prompt loop running somewhere that enforces reasoning.
huevosabio 3y ago
That's exactly how I feel about them. "Yes, and" to its logical conclusion.
__MatrixMan__ 3y ago
I find that if you create a context that allows for you to be wrong: > I am unfamilliar with foo, but my current belief is bar, although I must admit that baz is possible. I'm seeing evidence qux. Is my belief of bar about foo correct? Please explain in terms of qux. ...they will in fact contradict your position. I weasled my way past many prerequisties and am taking a science class that's way above my level. Being uncertain/humble with chatGPT has lead me to many lightbulb moments which were in contradiction to my initial beliefs, and have filled in my lack of prerequisites nicely. So you're not wrong, but it's a problem that's easy to avoid if you're careful about how you set up the context.
- ericjmorey 3y ago
  This isn't a contradiction to the improv model. It fits it perfectly. And being in a position not to be able to judge the output you receive by your own admission makes me question why you're so impressed with the output.
  - __MatrixMan__ 3y ago
    I'm impressed because when I verify it against the textbook it ends up being correct. I'd love to have time to just read the whole textbook, but ChatGPT uses words that appear in the index, so I end up reading only the parts I don't already know (twice, once as phrased by ChatGPT, a second time while verifying it). It feels like a better way to learn because you're not in passive-absorb-info mode for long periods. Instead you're in a much more active mode of alternating between asking questions, being critical of new info, and then confirm/denying it. Certainly not better than 1:1 instruction, but better than a lecture where there's only time to ask/answer two or three of your questions.

haswell 3y ago

> but at the end of the day they could just be madly improvising for all you know I think it's important to draw a line between the kind of madness and improvisation a human performs in these situations vs. an LLM. While it's still very possible that the LLM spits out something entirely wrong, the reasons for this wrongness are going to be quite different than the reasons a human performing improv will improvise something - anything - that keeps the improvisation going. The human is still fundamentally limited by the corpus of knowledge available to them, and by the limits of human cognition in processing all available information and generating a result. While I think the general point of thinking of an LLM as improvising vs. Googling is interesting to consider, I don't think it gets us closer to a useful mental model. A good LLM is still going to be right about most of the things it "improvises", even though it doesn't "know" anything. The person doing improv is not concerned about saying things that are factually true, and is focused on dynamically generating a story or joke or narrative based only on the immediate context. So while I do recognize some similarities, I don't think the similarities help us think about LLMs in a more useful way.

peteradio 3y ago
It sounds like you agree then and its only a matter of the corpus at hand. An improv actor would be called a comedian if they tried to interpolate based on spotty context.
- haswell 3y ago
  It’s a matter of the corpus at hand and the predictability of a human vs. a computer. With more training and improved algorithms, the computer can reach a high rate of accurate “improvisation”, at which point it doesn’t matter if it’s improv or not. My issue is with using improv to augment our mental models of LLMs, and I don’t agree that this is the end result. Taking this analogy too far misleads us about LLMs.
brucethemoose2 3y ago
> The person doing improv is not concerned about saying things that are factually true, and is focused on dynamically generating a story or joke or narrative based only on the immediate context. I disagree, as this is exactly what LLMs do! Its literally the function of a text completion transformers model. OpenAI has just "trained" their improv actor to speak like an an assistant and educated it well. They tell the model to roleplay an assistant in the script, and injects all sorts of facts and formatting into their script the user doesn't see. I believe they constrain grammar and post process it too. But ultimately the text is written by a heavily constrained improv actor that will absolutely improvise anything to keep the text flowing until the stop token arrives, even if truth from memory/context is usually the most likely improvisation.
- haswell 3y ago
  Except this is not exactly what LLMs do. This is quickly crossing into an anthropomorphism fallacy. The thing that matters most is how accurate (or not) the LLM is, regardless of how it arrived at its output. A human doing improv bears some surface level resemblance to an LLM processing text, but neither the context nor the mechanics are actually similar, nor are the theoretical limits of computing power. The broader point is that this mental model doesn’t help us better understand LLMs, and as LLMs continue to improve, any resemblance to “improv” will be less and less relevant (not to mention inaccurate).
  - brucethemoose2 3y ago
    I don't mean to anthropomorphize it, but I am trying to drive the point that current LLMs are text completion models. The current architectures continue a script, one token at a time... that's what it does, even if its a trillion trillion parameters and heavily aligned on all all accurate human communication ever. And that's similar enough to the basic mental loop of a "trained" improv actor to serve as a metaphor, I think.
    - haswell 3y ago
      > And that's similar enough to the basic mental loop of a "trained" improv actor My point is that this is only superficially similar, and to claim deeper similarity is the anthropomorphic fallacy. To form a useful mental model, the mechanics need to be similar enough to inform the person evaluating the system as the underlying system evolves. As LLMs evolve, the improv analogy becomes less and less useful because the LLM gets more and more accurate while the person doing improv is still just doing improv. The major difference being that people doing improv aren’t trying to be oracles of information. Perfecting the craft of improv may have nothing to do with accuracy, only believability or humor. More generally, thinking of LLMs as analogous to intelligent humans introduces other misconceptions, leads people to over-estimate accuracy, etc. The oddity at the bottom of all of this is that eventually, LLMs may be considered accurate enough to be used reliably as sources of information despite this “improvisation”, at which point it’s not really fair to call it improv, which would massively undersell the utility of the tool. But a perfectly accurate LLM may still not match any of the primary characteristics of an intelligent human, and it won’t matter as long as it’s useful.
recursive 3y ago
> A good LLM is still going to be right about most of the things it "improvises", even though it doesn't "know" anything. When I check on the factual details of what ChatGPT tells me, it seems to be wrong a lot of the time. Maybe not most, but I've stopped relying on it for factual information.
- haswell 3y ago
  I think this is a matter of framing. When compared to a system that does nothing but assembles grammatically correct sentences, the correctness of LLM output is impressively high. But ultimately I’m not saying that the accuracy of LLMs is sufficient, just that thinking about LLMs as improv machines isn’t very helpful.
  - seba_dos1 3y ago
    It has an encyclopedia around and a bit of human-scale time to consult it before coming up with the next line, but it's still more of an improv actor playing an expert than an actual expert. Sufficiently clever improv actor with sufficiently complete encyclopedia will at some point become indistinguishable from actual experts, but LLMs are still far from that point in general. They can already be great at fooling non-experts though, like an actor that needs to be just factually-correct enough to be believable in their role.

baq 3y ago

(In Soviet Russia, improv actors inject prompts into you.) LLMs are gullible, naive and will do anything to please you, because that's how they are trained. They're dogs who speak very good English and memoized the Internet.

blitz_skull 3y ago
I keep hearing these comparisons, and maybe on paper it’s “true” but if that’s the case; a significant portion of our actual problems can be exported to English dogs with sufficient memory. I regularly have hard problems that I can digest into chunks that LLMs routinely help me categorize, solve, and document. I feel like these comparisons are just not in touch with the actual amount of help that LLMs provide in a day to day context with programming related tasks.
- jdashg 3y ago
  It's true, most tasks are just simpler chunks of reasoning that anyone can do, tied together with key insights that are harder to train.
- efnx 3y ago
  Can you give an example because it has been my experience that they tend to give answers that sound right, but after closer inspection (or attempted compilation - or running) are in fact wrong.
  - baq 3y ago
    Dogs are obviously (I hope) a joke, actually they’re word calculators. They have some knowledge embedded because that’s needed to be able to be a word calculator but getting it facts in the context window works much better, hence they’re good at summarizing etc.
  - LeifCarrotson 3y ago
    They occasionally give answers that sound right but are in fact wrong. Their suitability for a particular task depends largely on your tolerance for that category of error. If you're generating fiction, it's irrelevant, if you're generating a newsletter that might be skimmed at a high level by disinterested people (or other LLMs) it's a low risk, if you're writing UI code it can be bad but will probably be corrected if it's bad enough, if you're writing critical code an insidious error can be disastrous.

all2 3y ago

My mental model of LLMs presently is as a "language core". You wouldn't trust your own stream of consciousness and you have checks and balances in your brain that you use to reason. In the same manner I don't really trust an LLM to "reason" unless there's a software/prompt loop running somewhere that enforces reasoning.

huevosabio 3y ago

That's exactly how I feel about them. "Yes, and" to its logical conclusion.

__MatrixMan__ 3y ago

I find that if you create a context that allows for you to be wrong: > I am unfamilliar with foo, but my current belief is bar, although I must admit that baz is possible. I'm seeing evidence qux. Is my belief of bar about foo correct? Please explain in terms of qux. ...they will in fact contradict your position. I weasled my way past many prerequisties and am taking a science class that's way above my level. Being uncertain/humble with chatGPT has lead me to many lightbulb moments which were in contradiction to my initial beliefs, and have filled in my lack of prerequisites nicely. So you're not wrong, but it's a problem that's easy to avoid if you're careful about how you set up the context.

ericjmorey 3y ago
This isn't a contradiction to the improv model. It fits it perfectly. And being in a position not to be able to judge the output you receive by your own admission makes me question why you're so impressed with the output.
- __MatrixMan__ 3y ago
  I'm impressed because when I verify it against the textbook it ends up being correct. I'd love to have time to just read the whole textbook, but ChatGPT uses words that appear in the index, so I end up reading only the parts I don't already know (twice, once as phrased by ChatGPT, a second time while verifying it). It feels like a better way to learn because you're not in passive-absorb-info mode for long periods. Instead you're in a much more active mode of alternating between asking questions, being critical of new info, and then confirm/denying it. Certainly not better than 1:1 instruction, but better than a lecture where there's only time to ask/answer two or three of your questions.