3 ms·

This is great, but it needs examples. I'll give you one: ChatGPT. Here's what the main page has to say: > We’ve trained a model called ChatGPT which interact

This is great, but it needs examples. I'll give you one: ChatGPT. Here's what the main page has to say: > We’ve trained a model called ChatGPT which interacts in a conversational way. The dialogue format makes it possible for ChatGPT to answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests. This is some careful writing. Most important are the weasel words, "possible for", and "in a...way". These make the context of the above paragraph ambiguous enough that it doesn't provide confidence in its own claims. But is that enough? In court, probably. To the market? It doesn't seem to be. It's clear that most of the conversation about ChatGPT is (ironically) a continuation of the same foundational misconceptions that are expressed (albeit without confidence) here. What is that misconception exactly? Identity. Here's one last claim that gets straight to the point: > While we’ve made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behavior. I want to focus on some very enlightening weasel words: "exhibit behavior". The more you read about what ChatGPT does, and more importantly, how it works, the more you are likely to see those magic words. That's because ChatGPT, from its most exciting results all the way down to its core behavior, doesn't actually behave. But isn't that its core feature? Aren't carefully trained behaviors the main thing we expect to see in its output? Yes! And we see them! How exciting. But there's a really important distinction that's really easy to miss: these behaviors are not ChatGPT's behaviors. They come from somewhere else entirely, and ChatGPT delivers them to us, usually at the right place and the right time. This is a case of mistaken identity. So what, then, does ChatGPT do exactly? It finds semantic continuations. Where? Inside the neutral network that models a suite of example semantics. It's only a model! So whose behavior is being exhibited? Language. Every exciting feature ChatGPT claims to provide is a feature of language itself. Interacts in a conversational way? Language. Answer follow-up questions? Language. Admit its mistakes? Language. Challenge incorrect premises, and reject inappropriate requests? Language. It's not just the exciting ones, either! It's the frustrating ones, too! Writes plausible-sounding but incorrect or nonsensical answers? The model is often excessively verbose and overuses certain phrases? All valid and common features of language. And one more limitation that really illustrates the misunderstanding about what ChatGPT does: > given one phrasing of a question, the model can claim to not know the answer, but given a slight rephrase, can answer correctly. That's 100% semantically valid language. Language does nothing to stop us from telling lies! When ChatGPT pulls a continuation out of its model, there may be multiple options that are both valid, and contradictory! It might pull the reactionary response, "I don't know" (a very common continuation in real speech to any sentence structured as a question) or it might pull a domain-specific answer that responds to the subjects in that sentence. It would be really useful if we could just teach it to do the second one, right? But we can't, because ChatGPT never draws a distinction between grammar and subjects. In fact, ChatGPT doesn't even distinguish words or punctuation! ChatGPT can never understand. There's no place in its behavior to introduce logic. The best it can do is practice. Instead of teaching ChatGPT, its authors give it carefully filtered data to model, and then put their fingers on the scale to make continuations they prefer more likely to be chosen. In a word: training. That's the best that can be done. Anything more requires a different kind of tool entirely.

nullc 4y ago
> This is some careful writing. Most important are the weasel words, So what do you want, a full refund? OpenAI is probably pretty busy, so I'll handle that for them and give you your nothing back. Glad to be of service. :P > It would be really useful if we could just teach it to do the second one, right? But we can't, because ChatGPT never draws a distinction between grammar and subjects. The structure of the world, as expressed in its training data, is part of it's 'grammar'. It absolute can and does favor giving truthful answers even though a lie is equally grammatical English. One only need spend a minute looking at HN though to see that "I don't know" is really not part of the language used in most places online, so far they haven't come up with a way of training the model on all the places where someone saw something and decided to say nothing. :) > In a word: training. We all also learn about the world through training. The LLMs aren't at all like human minds-- they're fixed depth circuits and can only engage in longer thoughts by 'thinking out loud', as an example. But to say it can't distinguish truth from lies because both are valid language sounds like an error resulting from hearing the title "language model". LLMs model the world through language.
- thomastjeffery 4y ago
  I want people to understand what the thing is, and what it isn't. By people, I especially mean the people involved in making and selling it. The overwhelming majority of talk I see on the subject misunderstands this very important distinction. Because it is missed, people are taking about ChatGPT personified. They are talking about the version that lives in their hopes and dreams, as if that version is just around the corner. It isn't. It just appears to be. Our hopes and dreams are the content on exhibition. We see them echoed in semantic reconstruction, but what we are looking at is not the work of a new writer: it's the old words of many, successfully repurposed in new semantic space. It's as if we took all the stories we love, and split them into puzzle pieces, then fit the pieces together in a new order. Because language is so flexible, many of them fit, and even look good together. Language is so good that it pre-sorts subjects, objects, phrases, responses, and even logic. The puzzle-maker doesn't even have to know what they are working with to cut the pieces in a useful way! > It absolute can and does favor giving truthful answers even though a lie is equally grammatical English. That depends on the context. In cases that it has the answer, that is mostly true. This covers a lot of cases, especially those that are likely to be tested; because many such cases have "failed" and been resolved. That's literally what testing looks like. But it's still enough of an open problem that they write about it on the main page. The very fact that OpenAI has managed to put the sheer quantity and quality of behavior into their model that they have, is indeed impressive. But it is those people who deserve the credit, no matter how much they want to attribute that credit to ChatGPT itself. My point here is that every "desirable answer" - that ChatGPT provides as a continuation - comes from the already-existing language in its training data. It may be semantically constructed as a result of being pulled from a semantic model, but we need to stop pretending it is symbolically constructed. That work happens entirely in the construction and application of the training data itself. ChatGPT is never doing that part of the job. > One only need spend a minute looking at HN though to see that "I don't know" is really not part of the language used in most places online Exactly! Every time any of the "places online" (that end up used as training data) exhibit this behavior, that is an instance of the work being done. The finished product of that very work is freely available. It only needs to be found in one of several semantically correct and semantically related contexts. That's what ChatGPT does, and nothing more. By virtue of the average given persons' interest in detailed answers to questions, and the average persons' dislike for conversations that end in non-answers, we are already doing the work. Just by participating in the most common internet dialogues, we are creating a wealth of data that captures the same behavior we hope to see in ChatGPT's continuations. > they're fixed depth circuits and can only engage in longer thoughts by 'thinking out loud', as an example. That's a great summary of my point. Why are you arguing if you agree with me? The phrase, "thinking out loud" is an excellent illustration for the power of language itself to capture and encapsulate logical behavior. We could be spending a lot of time constructing semantically valid nonsense like, "colorless green ideas sleep furiously". But any time you see that sentence, you are likely to see it surrounded in thoughtful context. This sentence is a popular example, not only of the ability for language to contain nonsense, but also for our tendency as language users to avoid doing just that. A story isn't just a jumble of semantically valid symbols: there is meaning expressed. We don't write a lot of nonsense. We do write a lot of meaning. That meaning is neatly packaged into the semantics of language. To find it, one only needs to unwrap the semantics themselves. But that only works when the meaning has been written somewhere. And there is no way to distinguish between two conflicting meanings that live in the same semantic space. Even if neither one is a lie, the semantics must disambiguate, or one of the options has to be preferred as a continuation to its expected context; during the training step. > LLMs model the world through language. They don't model the world: people do. They just model the language. Because we have packaged our models of the world (and every interaction with those models) all together into language, we can see language models echo all those behaviors back to us.

nullc 4y ago

> This is some careful writing. Most important are the weasel words, So what do you want, a full refund? OpenAI is probably pretty busy, so I'll handle that for them and give you your nothing back. Glad to be of service. :P > It would be really useful if we could just teach it to do the second one, right? But we can't, because ChatGPT never draws a distinction between grammar and subjects. The structure of the world, as expressed in its training data, is part of it's 'grammar'. It absolute can and does favor giving truthful answers even though a lie is equally grammatical English. One only need spend a minute looking at HN though to see that "I don't know" is really not part of the language used in most places online, so far they haven't come up with a way of training the model on all the places where someone saw something and decided to say nothing. :) > In a word: training. We all also learn about the world through training. The LLMs aren't at all like human minds-- they're fixed depth circuits and can only engage in longer thoughts by 'thinking out loud', as an example. But to say it can't distinguish truth from lies because both are valid language sounds like an error resulting from hearing the title "language model". LLMs model the world through language.

thomastjeffery 4y ago
I want people to understand what the thing is, and what it isn't. By people, I especially mean the people involved in making and selling it. The overwhelming majority of talk I see on the subject misunderstands this very important distinction. Because it is missed, people are taking about ChatGPT personified. They are talking about the version that lives in their hopes and dreams, as if that version is just around the corner. It isn't. It just appears to be. Our hopes and dreams are the content on exhibition. We see them echoed in semantic reconstruction, but what we are looking at is not the work of a new writer: it's the old words of many, successfully repurposed in new semantic space. It's as if we took all the stories we love, and split them into puzzle pieces, then fit the pieces together in a new order. Because language is so flexible, many of them fit, and even look good together. Language is so good that it pre-sorts subjects, objects, phrases, responses, and even logic. The puzzle-maker doesn't even have to know what they are working with to cut the pieces in a useful way! > It absolute can and does favor giving truthful answers even though a lie is equally grammatical English. That depends on the context. In cases that it has the answer, that is mostly true. This covers a lot of cases, especially those that are likely to be tested; because many such cases have "failed" and been resolved. That's literally what testing looks like. But it's still enough of an open problem that they write about it on the main page. The very fact that OpenAI has managed to put the sheer quantity and quality of behavior into their model that they have, is indeed impressive. But it is those people who deserve the credit, no matter how much they want to attribute that credit to ChatGPT itself. My point here is that every "desirable answer" - that ChatGPT provides as a continuation - comes from the already-existing language in its training data. It may be semantically constructed as a result of being pulled from a semantic model, but we need to stop pretending it is symbolically constructed. That work happens entirely in the construction and application of the training data itself. ChatGPT is never doing that part of the job. > One only need spend a minute looking at HN though to see that "I don't know" is really not part of the language used in most places online Exactly! Every time any of the "places online" (that end up used as training data) exhibit this behavior, that is an instance of the work being done. The finished product of that very work is freely available. It only needs to be found in one of several semantically correct and semantically related contexts. That's what ChatGPT does, and nothing more. By virtue of the average given persons' interest in detailed answers to questions, and the average persons' dislike for conversations that end in non-answers, we are already doing the work. Just by participating in the most common internet dialogues, we are creating a wealth of data that captures the same behavior we hope to see in ChatGPT's continuations. > they're fixed depth circuits and can only engage in longer thoughts by 'thinking out loud', as an example. That's a great summary of my point. Why are you arguing if you agree with me? The phrase, "thinking out loud" is an excellent illustration for the power of language itself to capture and encapsulate logical behavior. We could be spending a lot of time constructing semantically valid nonsense like, "colorless green ideas sleep furiously". But any time you see that sentence, you are likely to see it surrounded in thoughtful context. This sentence is a popular example, not only of the ability for language to contain nonsense, but also for our tendency as language users to avoid doing just that. A story isn't just a jumble of semantically valid symbols: there is meaning expressed. We don't write a lot of nonsense. We do write a lot of meaning. That meaning is neatly packaged into the semantics of language. To find it, one only needs to unwrap the semantics themselves. But that only works when the meaning has been written somewhere. And there is no way to distinguish between two conflicting meanings that live in the same semantic space. Even if neither one is a lie, the semantics must disambiguate, or one of the options has to be preferred as a continuation to its expected context; during the training step. > LLMs model the world through language. They don't model the world: people do. They just model the language. Because we have packaged our models of the world (and every interaction with those models) all together into language, we can see language models echo all those behaviors back to us.