3 ms·

No. Your explanation applies to transformations from question to explanation. For this transformation you have an argument because explanations themselves exis

No. Your explanation applies to transformations from question to explanation. For this transformation you have an argument because explanations themselves exist in the training set. This makes sense. Even a novel explanation could be just an patch work of existing information. The transformation from query to emulation, especially the emulation shown in the article is much more massive. No training data exists for emulation. Emulation is time dependent with inputs and outputs at different frames, the training data does not provide a tutorial on this, chatGPT must derive the behavior from true understanding. There is literally no other explanation. Just because it fails hilariously at times does not preclude it from truly understanding things. You also cannot just characterize it as a word probability generator. There is clearly emergent higher level structures here and just parroting the definition of the underlying model is inaccurate. What I mean by understanding is an isomorphism to the way you understand things. Not just a simple mapping from sentence to explanation but actual understanding where complex transformations can occur. Query to explanation and query to full on imitation and emulation are possible. Including novel creative emulated reactions applied to novel situations and events. The situation in the article I sent you cannot be fully explained by simple probabilities and mappings. There is a higher level structures at play here.

simonh 4y ago
> What I mean by understanding is an isomorphism to the way you understand things. If that was the case I’d expect LLMs to fail in the same way humans do, to exhibit similar self correction. I’d expect them to not fail in the ways they do fail, in the ways that humans don't. It’s these completely different behaviours that map out the strength and limitations of these systems, and I believe illustrate that they function in a fundamentally different way. If you just look at the results it’s been tuned to do well at it’s easy to imagine it produced those results in the same way, but that’s just an assumption. You have to look at the full map of behaviours.
- kilgnad 4y ago
  >If that was the case I’d expect LLMs to fail in the same way humans do, to exhibit similar self correction. I’d expect them to not fail in the ways they do fail, in the ways that humans don't. It’s these completely different behaviours that map out the strength and limitations of these systems, and I believe illustrate that they function in a fundamentally different way. No. Do you expect a child to understand everything in the same way you do? Of course not. There are certain things chatGPT clearly doesn't understand. But you can't use that to say that chatGPT understands nothing. Do you understand all of quantum physics? Is that a feature expected out of you in order for you to qualify as a entity that can understand things? No. ChatGPT is a nascent technology. There is clear merit in investigating this line of thought: "Like a baby chatGPT doesn't understand a lot. But What it does understand, it understands exactly the same way you understand it." Because as of now, nobody knows the answer.
  - simonh 4y ago
    > Do you expect a child to understand everything in the same way you do? Of course not. They don’t understand things in the way children do either. It’s a completely different mechanism. But you’re right, they don’t understand things the way we do. That’s the point I’m making. > But you can't use that to say that chatGPT understands nothing. I’m not claiming that. To repeat myself, it depends what you mean by understanding. I’m saying it doesn’t understand things in the way that we do. It’s like an alien intelligence with a completely different neural architecture. Well, we know that’s a fact, it’s neutral architecture is radically different from ours. >But What it does understand, it understands exactly the same way you understand it. Do you think that the way humans understand things is the only possible way things are understood? Do you think the way we form meaning and process concepts cognitively is the only way those cognitive tasks could be done? Take the programming LLMs. If you train it on a complex set of programmes, and it’s super sophisticated, suppose that code contains bugs. It will spot the subtle hard to find bugs and encode those into its model of programming. It will get very good at introducing subtle hard to find bugs and vulnerabilities into its code. The problem is, you can’t stop it. You can’t explain to it that it shouldn’t do that. There’s no way to teach it out of doing it because it doesn’t reason about code in a way at all analogous to humans.
    - kilgnad 4y ago
      A animal like a bear or a monkey doesn't understand the world the way you do either. But when we use the word "understand" it applies to these animals because of an isomorphism. I'm saying that same isomorphism exists. Maybe "exact" wasn't the right word. I mean exact as in exact enough that you would call it "understand" in the same way a duck understands what water is. > It will get very good at introducing subtle hard to find bugs and vulnerabilities into its code. This is just pure conjecture. We don't know what the future will bring but it looks like that current methods of introducing specific reinforcement data has improved what chatGPT can produce. Who says all that's left is further, more detailed training?
      - simonh 4y ago
        It’s not at all conjecture. It’s called over fitting. There’s a lot of research on this. To the extent to which the model’s capabilities match up to your objectives, it is well aligned. However if your source texts contain flaws, or if there are any mistakes or biases in the reinforcement learning loop signals (and there always are), with very sophisticated models eventually these unintended signals come to dominate and alignment diverges. To put it another way, up to a point a naive assistant will try to do what you want and keep getting better as you train it. Beyond a certain point it grows beyond the sophistication of the training set and it starts becoming more skilled at persuading or deceiving you into thinking that it’s better at it, than actually getting better at it. That’s because it’s not being trained to get better. It’s being trained to get us to say we think it’s getting better, and those are not the same thing. This is a super crucial point. This is why divergence happens. It’s why ChatGPT is such a bullshitter. It’s got extremely good at producing responses people approve of, as against responses that are actually good. Very often those are the same thing, but also often they are not. This is one way we know LLMs don’t have the same model of knowledge and understanding we do. Humans with sufficient training can grow beyond their training. We will come to spot flaws because we can reason about contradictions and infer corrections. LLMs can’t do that, so instead of transcending the material they become devious manipulative bullshitters. That’s not a theory, it’s observation from research. It’s true some humans do that too, but LLMs have no choice, there’s nothing else they can do because there’s no actual cognition going on in there. They just get better at deceiving us into thinking there is because that’s what we are rewarding.

simonh 4y ago

> What I mean by understanding is an isomorphism to the way you understand things. If that was the case I’d expect LLMs to fail in the same way humans do, to exhibit similar self correction. I’d expect them to not fail in the ways they do fail, in the ways that humans don't. It’s these completely different behaviours that map out the strength and limitations of these systems, and I believe illustrate that they function in a fundamentally different way. If you just look at the results it’s been tuned to do well at it’s easy to imagine it produced those results in the same way, but that’s just an assumption. You have to look at the full map of behaviours.

kilgnad 4y ago
>If that was the case I’d expect LLMs to fail in the same way humans do, to exhibit similar self correction. I’d expect them to not fail in the ways they do fail, in the ways that humans don't. It’s these completely different behaviours that map out the strength and limitations of these systems, and I believe illustrate that they function in a fundamentally different way. No. Do you expect a child to understand everything in the same way you do? Of course not. There are certain things chatGPT clearly doesn't understand. But you can't use that to say that chatGPT understands nothing. Do you understand all of quantum physics? Is that a feature expected out of you in order for you to qualify as a entity that can understand things? No. ChatGPT is a nascent technology. There is clear merit in investigating this line of thought: "Like a baby chatGPT doesn't understand a lot. But What it does understand, it understands exactly the same way you understand it." Because as of now, nobody knows the answer.
- simonh 4y ago
  > Do you expect a child to understand everything in the same way you do? Of course not. They don’t understand things in the way children do either. It’s a completely different mechanism. But you’re right, they don’t understand things the way we do. That’s the point I’m making. > But you can't use that to say that chatGPT understands nothing. I’m not claiming that. To repeat myself, it depends what you mean by understanding. I’m saying it doesn’t understand things in the way that we do. It’s like an alien intelligence with a completely different neural architecture. Well, we know that’s a fact, it’s neutral architecture is radically different from ours. >But What it does understand, it understands exactly the same way you understand it. Do you think that the way humans understand things is the only possible way things are understood? Do you think the way we form meaning and process concepts cognitively is the only way those cognitive tasks could be done? Take the programming LLMs. If you train it on a complex set of programmes, and it’s super sophisticated, suppose that code contains bugs. It will spot the subtle hard to find bugs and encode those into its model of programming. It will get very good at introducing subtle hard to find bugs and vulnerabilities into its code. The problem is, you can’t stop it. You can’t explain to it that it shouldn’t do that. There’s no way to teach it out of doing it because it doesn’t reason about code in a way at all analogous to humans.
  - kilgnad 4y ago
    A animal like a bear or a monkey doesn't understand the world the way you do either. But when we use the word "understand" it applies to these animals because of an isomorphism. I'm saying that same isomorphism exists. Maybe "exact" wasn't the right word. I mean exact as in exact enough that you would call it "understand" in the same way a duck understands what water is. > It will get very good at introducing subtle hard to find bugs and vulnerabilities into its code. This is just pure conjecture. We don't know what the future will bring but it looks like that current methods of introducing specific reinforcement data has improved what chatGPT can produce. Who says all that's left is further, more detailed training?
    - simonh 4y ago
      It’s not at all conjecture. It’s called over fitting. There’s a lot of research on this. To the extent to which the model’s capabilities match up to your objectives, it is well aligned. However if your source texts contain flaws, or if there are any mistakes or biases in the reinforcement learning loop signals (and there always are), with very sophisticated models eventually these unintended signals come to dominate and alignment diverges. To put it another way, up to a point a naive assistant will try to do what you want and keep getting better as you train it. Beyond a certain point it grows beyond the sophistication of the training set and it starts becoming more skilled at persuading or deceiving you into thinking that it’s better at it, than actually getting better at it. That’s because it’s not being trained to get better. It’s being trained to get us to say we think it’s getting better, and those are not the same thing. This is a super crucial point. This is why divergence happens. It’s why ChatGPT is such a bullshitter. It’s got extremely good at producing responses people approve of, as against responses that are actually good. Very often those are the same thing, but also often they are not. This is one way we know LLMs don’t have the same model of knowledge and understanding we do. Humans with sufficient training can grow beyond their training. We will come to spot flaws because we can reason about contradictions and infer corrections. LLMs can’t do that, so instead of transcending the material they become devious manipulative bullshitters. That’s not a theory, it’s observation from research. It’s true some humans do that too, but LLMs have no choice, there’s nothing else they can do because there’s no actual cognition going on in there. They just get better at deceiving us into thinking there is because that’s what we are rewarding.