3 ms·

> I think that you are agreeing with the main point I'm trying to make, and you're even going further than me. Pretty much. > You're just disagreeing with my

> I think that you are agreeing with the main point I'm trying to make, and you're even going further than me. Pretty much. > You're just disagreeing with my wording. Yeah, that's the "going even further" part. Words are important. The importance of words is really foundational for a lot of what we are talking about. > What you're describing is, it doesn't think, it doesn't understand, it just follows whatever patterns in its training corpus kind of correlate with patterns in the prompt. And I agree. Cool. It's nice to be on the same page. Let's read the next one: > Where I may differ is how to regard the training corpus. True, it contains a massive amount of distilled thought, encoded in words. That's the biggest point I'm trying to make by "disagreeing with your wording". Words have significant implications. In this case, the implication is "problem domain": language vs. text. Language is a subdomain of written text. Written text is a subdomain of all possible text. GPT is dealing with written text. GPT does not only model the language patterns that humans intentionally put into text. GPT models all of the patterns that exist in the text. Humans, when writing, do "distill" thought into language; but the result is messy: we end up encoding more patterns than we intended to. Our behavior, our choices, our reasons, and the surrounding situation we exist in, all inform our writing. Those are the patterns that get encoded into the text, right alongside language. GPT doesn't see any pattern as different from another; so it's misleading to say that it models "language". GPT doesn't categorize that specific set of patterns: it models all patterns equally. We humans have a good idea of what to expect from only language patterns. If that were all that was modeled, GPT's output wouldn't be nearly as interesting to us. GPT's output is interesting to us. If we look objectively at the model it generates, we can see why: there are patterns there we had no intention of writing; and GPT modeled them just like it modeled the language-specific patterns we expected it to.

AnimalMuppet 4y ago
> GPT does not only model the language patterns that humans intentionally put into text. GPT models all of the patterns that exist in the text. > Humans, when writing, do "distill" thought into language; but the result is messy: we end up encoding more patterns than we intended to. Our behavior, our choices, our reasons, and the surrounding situation we exist in, all inform our writing. Those are the patterns that get encoded into the text, right alongside language. Sometimes truly, and sometimes falsely. Take this paragraph, for instance. If I made a typo on the fifth word, and left out the "l", this paragraph would also encode something that is not at all what it is about. A human reader might chuckle at the typo, but would not be confused by the meaning. But for GPT, that will be a pattern in the text, just like all the others. So sometimes it can reveal what was unintended but is still informative, and sometimes it can be misleading. But it's not just typos. There's very intentional text from advertisers, shill, liars, and propagandists. That is all patterns that get encoded, too. The lies are encoded right along with the truth. But where I was really going initially was the hallucinations. If I ask GPT who won this year's NASCAR race on Tierra del Fuego, it will find something - some pattern that matches better than other patterns. Who knows, it might find some kid's fanfic posted on Reddit, and will tell me who won in the kid's story. But if there's nothing like that, it will find something. It won't be any kind of a meaningful pattern that it finds, but it will still present it, and will present it with as much certainty as it would if it found a number of news stories that all agreed with each other. So I'm seeing in your posts a pattern encoded (perhaps falsely) that makes me think that you think that what the text encodes is gospel. It's not. It's much that is golden and much that is dross, and GIGO applies to GPT as much as any other information system.
- thomastjeffery 4y ago
  > So I'm seeing in your posts a pattern encoded (perhaps falsely) that makes me think that you think that what the text encodes is gospel. On the contrary. I think you have a pretty good description for how the behavior we want and the behavior we don't want exist as the same features. I am trying to restructure the narrative around LLMs to be about the patterns themselves, not what we hope the patterns to be. We can throw around the word, "hallucinate", but that's just another anthropomorphization. Whether or not the pattern followed "makes sense" is a categorization humans attribute after the fact. GPT doesn't do "certainty". It does what it is given.
  - AnimalMuppet 4y ago
    OK. I think I understand, and I think I agree. Thanks for the conversation.
    - thomastjeffery 4y ago
      You're welcome! I've been finding a certain satisfaction in trying to articulate this perspective. It's challenging because I'm criticizing most of the narrative that exists in the subject, but that narrative also gives a clear illustration of where to go wrong. I think better understanding the more mysterious patterns that exist in our writing might be really useful. Often, the first step in inventing something useful is to explore an unfamiliar domain and recognize what information we are missing.

AnimalMuppet 4y ago

> GPT does not only model the language patterns that humans intentionally put into text. GPT models all of the patterns that exist in the text. > Humans, when writing, do "distill" thought into language; but the result is messy: we end up encoding more patterns than we intended to. Our behavior, our choices, our reasons, and the surrounding situation we exist in, all inform our writing. Those are the patterns that get encoded into the text, right alongside language. Sometimes truly, and sometimes falsely. Take this paragraph, for instance. If I made a typo on the fifth word, and left out the "l", this paragraph would also encode something that is not at all what it is about. A human reader might chuckle at the typo, but would not be confused by the meaning. But for GPT, that will be a pattern in the text, just like all the others. So sometimes it can reveal what was unintended but is still informative, and sometimes it can be misleading. But it's not just typos. There's very intentional text from advertisers, shill, liars, and propagandists. That is all patterns that get encoded, too. The lies are encoded right along with the truth. But where I was really going initially was the hallucinations. If I ask GPT who won this year's NASCAR race on Tierra del Fuego, it will find something - some pattern that matches better than other patterns. Who knows, it might find some kid's fanfic posted on Reddit, and will tell me who won in the kid's story. But if there's nothing like that, it will find something. It won't be any kind of a meaningful pattern that it finds, but it will still present it, and will present it with as much certainty as it would if it found a number of news stories that all agreed with each other. So I'm seeing in your posts a pattern encoded (perhaps falsely) that makes me think that you think that what the text encodes is gospel. It's not. It's much that is golden and much that is dross, and GIGO applies to GPT as much as any other information system.

thomastjeffery 4y ago
> So I'm seeing in your posts a pattern encoded (perhaps falsely) that makes me think that you think that what the text encodes is gospel. On the contrary. I think you have a pretty good description for how the behavior we want and the behavior we don't want exist as the same features. I am trying to restructure the narrative around LLMs to be about the patterns themselves, not what we hope the patterns to be. We can throw around the word, "hallucinate", but that's just another anthropomorphization. Whether or not the pattern followed "makes sense" is a categorization humans attribute after the fact. GPT doesn't do "certainty". It does what it is given.
- AnimalMuppet 4y ago
  OK. I think I understand, and I think I agree. Thanks for the conversation.
  - thomastjeffery 4y ago
    You're welcome! I've been finding a certain satisfaction in trying to articulate this perspective. It's challenging because I'm criticizing most of the narrative that exists in the subject, but that narrative also gives a clear illustration of where to go wrong. I think better understanding the more mysterious patterns that exist in our writing might be really useful. Often, the first step in inventing something useful is to explore an unfamiliar domain and recognize what information we are missing.