4 ms·

I’ve read that before, it is fascinating. GPT3 was just trained on source text, but ChatGPT has undergone an an additional very rigorous programme of intensive

by simonh 4y ago

I’ve read that before, it is fascinating. GPT3 was just trained on source text, but ChatGPT has undergone an an additional very rigorous programme of intensive training bootstrapped by human feedback. This has evolved it in order to improve its performance as an information assistant, especially on specific tasks that have proved problematic in the past. One of the things GPT3 has been criticised for is not understanding about people, so you can bet your bottom dollar one of the specific things it’s been coached on is people and how humans work. But it’s still just a language model. It’s just that now it’s been coached how to splurge out better text about people. The underlying limitations of the architecture are still there though. Take the way until recently if you asked it to respond in Danish, it would reply in perfect Danish about how it was an English language model and could not provide responses in Danish. When pointed out that it was replying in Danish, it would adamantly deny it, in Danish. That’s been fixed now. It’s totally mindless, it’s just being coached to provide better and better responses to hide its mindlessness. But papering over the cracks like this with flaw specific training isn’t creating a mind, because it all still works in the same way. This video explains the approach and some of its limitations. https://youtu.be/viJt_DXTfwA https://youtu.be/viJt_DXTfwA

kilgnad 4y ago
Except in the example that you read, it cannot have done what it did without understanding what it was asked.
- simonh 4y ago
  It depends exactly what you mean by understanding. It has a map of connections between word tokens that allow it to generate an output of word tokens that are useful to us, and which has now just recently been specifically trained for this particular problem domain. So now for these sorts of questions it no longer produces gibberish. None of that works in a way at all analogous to how the human brain processes language, reasons about concepts, or connects ideas beyond just words. It’s not meaning in the sense that we generally understand meaning. If you take a step outside its trained response envelope, it will still fail hilariously generating meaningless drivel. It’s grasp of concepts just isn’t there, it has no concrete foundation. All it knows is word frequency weightings. Don’t get me wrong it’s amazing engineering, it’s probably going to be incredibly useful.
  - kilgnad 4y ago
    No. Your explanation applies to transformations from question to explanation. For this transformation you have an argument because explanations themselves exist in the training set. This makes sense. Even a novel explanation could be just an patch work of existing information. The transformation from query to emulation, especially the emulation shown in the article is much more massive. No training data exists for emulation. Emulation is time dependent with inputs and outputs at different frames, the training data does not provide a tutorial on this, chatGPT must derive the behavior from true understanding. There is literally no other explanation. Just because it fails hilariously at times does not preclude it from truly understanding things. You also cannot just characterize it as a word probability generator. There is clearly emergent higher level structures here and just parroting the definition of the underlying model is inaccurate. What I mean by understanding is an isomorphism to the way you understand things. Not just a simple mapping from sentence to explanation but actual understanding where complex transformations can occur. Query to explanation and query to full on imitation and emulation are possible. Including novel creative emulated reactions applied to novel situations and events. The situation in the article I sent you cannot be fully explained by simple probabilities and mappings. There is a higher level structures at play here.
    - simonh 4y ago
      > What I mean by understanding is an isomorphism to the way you understand things. If that was the case I’d expect LLMs to fail in the same way humans do, to exhibit similar self correction. I’d expect them to not fail in the ways they do fail, in the ways that humans don't. It’s these completely different behaviours that map out the strength and limitations of these systems, and I believe illustrate that they function in a fundamentally different way. If you just look at the results it’s been tuned to do well at it’s easy to imagine it produced those results in the same way, but that’s just an assumption. You have to look at the full map of behaviours.
      - kilgnad 4y ago
        >If that was the case I’d expect LLMs to fail in the same way humans do, to exhibit similar self correction. I’d expect them to not fail in the ways they do fail, in the ways that humans don't. It’s these completely different behaviours that map out the strength and limitations of these systems, and I believe illustrate that they function in a fundamentally different way. No. Do you expect a child to understand everything in the same way you do? Of course not. There are certain things chatGPT clearly doesn't understand. But you can't use that to say that chatGPT understands nothing. Do you understand all of quantum physics? Is that a feature expected out of you in order for you to qualify as a entity that can understand things? No. ChatGPT is a nascent technology. There is clear merit in investigating this line of thought: "Like a baby chatGPT doesn't understand a lot. But What it does understand, it understands exactly the same way you understand it." Because as of now, nobody knows the answer.

by simonh 4y ago

kilgnad 4y ago

Except in the example that you read, it cannot have done what it did without understanding what it was asked.

simonh 4y ago
It depends exactly what you mean by understanding. It has a map of connections between word tokens that allow it to generate an output of word tokens that are useful to us, and which has now just recently been specifically trained for this particular problem domain. So now for these sorts of questions it no longer produces gibberish. None of that works in a way at all analogous to how the human brain processes language, reasons about concepts, or connects ideas beyond just words. It’s not meaning in the sense that we generally understand meaning. If you take a step outside its trained response envelope, it will still fail hilariously generating meaningless drivel. It’s grasp of concepts just isn’t there, it has no concrete foundation. All it knows is word frequency weightings. Don’t get me wrong it’s amazing engineering, it’s probably going to be incredibly useful.
- kilgnad 4y ago
  No. Your explanation applies to transformations from question to explanation. For this transformation you have an argument because explanations themselves exist in the training set. This makes sense. Even a novel explanation could be just an patch work of existing information. The transformation from query to emulation, especially the emulation shown in the article is much more massive. No training data exists for emulation. Emulation is time dependent with inputs and outputs at different frames, the training data does not provide a tutorial on this, chatGPT must derive the behavior from true understanding. There is literally no other explanation. Just because it fails hilariously at times does not preclude it from truly understanding things. You also cannot just characterize it as a word probability generator. There is clearly emergent higher level structures here and just parroting the definition of the underlying model is inaccurate. What I mean by understanding is an isomorphism to the way you understand things. Not just a simple mapping from sentence to explanation but actual understanding where complex transformations can occur. Query to explanation and query to full on imitation and emulation are possible. Including novel creative emulated reactions applied to novel situations and events. The situation in the article I sent you cannot be fully explained by simple probabilities and mappings. There is a higher level structures at play here.
  - simonh 4y ago
    > What I mean by understanding is an isomorphism to the way you understand things. If that was the case I’d expect LLMs to fail in the same way humans do, to exhibit similar self correction. I’d expect them to not fail in the ways they do fail, in the ways that humans don't. It’s these completely different behaviours that map out the strength and limitations of these systems, and I believe illustrate that they function in a fundamentally different way. If you just look at the results it’s been tuned to do well at it’s easy to imagine it produced those results in the same way, but that’s just an assumption. You have to look at the full map of behaviours.
    - kilgnad 4y ago
      >If that was the case I’d expect LLMs to fail in the same way humans do, to exhibit similar self correction. I’d expect them to not fail in the ways they do fail, in the ways that humans don't. It’s these completely different behaviours that map out the strength and limitations of these systems, and I believe illustrate that they function in a fundamentally different way. No. Do you expect a child to understand everything in the same way you do? Of course not. There are certain things chatGPT clearly doesn't understand. But you can't use that to say that chatGPT understands nothing. Do you understand all of quantum physics? Is that a feature expected out of you in order for you to qualify as a entity that can understand things? No. ChatGPT is a nascent technology. There is clear merit in investigating this line of thought: "Like a baby chatGPT doesn't understand a lot. But What it does understand, it understands exactly the same way you understand it." Because as of now, nobody knows the answer.