5 ms·

A animal like a bear or a monkey doesn't understand the world the way you do either. But when we use the word "understand" it applies to these animals because o

A animal like a bear or a monkey doesn't understand the world the way you do either. But when we use the word "understand" it applies to these animals because of an isomorphism. I'm saying that same isomorphism exists. Maybe "exact" wasn't the right word. I mean exact as in exact enough that you would call it "understand" in the same way a duck understands what water is. > It will get very good at introducing subtle hard to find bugs and vulnerabilities into its code. This is just pure conjecture. We don't know what the future will bring but it looks like that current methods of introducing specific reinforcement data has improved what chatGPT can produce. Who says all that's left is further, more detailed training?

simonh 4y ago
It’s not at all conjecture. It’s called over fitting. There’s a lot of research on this. To the extent to which the model’s capabilities match up to your objectives, it is well aligned. However if your source texts contain flaws, or if there are any mistakes or biases in the reinforcement learning loop signals (and there always are), with very sophisticated models eventually these unintended signals come to dominate and alignment diverges. To put it another way, up to a point a naive assistant will try to do what you want and keep getting better as you train it. Beyond a certain point it grows beyond the sophistication of the training set and it starts becoming more skilled at persuading or deceiving you into thinking that it’s better at it, than actually getting better at it. That’s because it’s not being trained to get better. It’s being trained to get us to say we think it’s getting better, and those are not the same thing. This is a super crucial point. This is why divergence happens. It’s why ChatGPT is such a bullshitter. It’s got extremely good at producing responses people approve of, as against responses that are actually good. Very often those are the same thing, but also often they are not. This is one way we know LLMs don’t have the same model of knowledge and understanding we do. Humans with sufficient training can grow beyond their training. We will come to spot flaws because we can reason about contradictions and infer corrections. LLMs can’t do that, so instead of transcending the material they become devious manipulative bullshitters. That’s not a theory, it’s observation from research. It’s true some humans do that too, but LLMs have no choice, there’s nothing else they can do because there’s no actual cognition going on in there. They just get better at deceiving us into thinking there is because that’s what we are rewarding.
- kilgnad 4y ago
  You can't say the entire curve is over-fitted. The space is vast and the training data is sparse so given the amount it has to cover, for sure segments of the curve must be over-fitted and other segments of the curve must be more generalized. Additionally you can't say that just because artificial neural nets have a property of "over-fitting" doesn't mean that humans themselves can't be in a state where they are themselves over-fitted. So right now given the emergent properties of LLMs we can't fully delineate the concept of human understanding away from what the AI is doing.
  - simonh 4y ago
    I did not say the entire curve is over fitted, I said that in some circumstances with highly sophisticated models you get over fitting. Humans over fit in different ways. A human who saw security code that introduced a buffer overflow bug that made it vulnerable to attack might make the mistake of implementing new code in a similar way and introducing a similar bug. The human isn’t deliberately introducing bugs, they didn’t spot the bug. When an LLM over fits the point is it does spot the bug. Because the input training programs define the goal, introducing bugs like that becomes one of the goals for the LLM. This is a consequence of the different way LLMs encode knowledge. More technically, it’s the different ways humans and LLMs infer goals, which is an important aspect of it. Anyway this has become a long thread, much appreciated. I’ll just summarise by saying it seems like there are many radically different and perhaps infinite possible ways question response could be implemented. Just because the surface responses these things produce in some ways seem analogous to the responses humans provide in many cases really shouldn’t be taken as evidence they perform them the same way we do. Especially given their neural architecture is radically different from ours. So how about we say we’ll keep open minds about the future development of this technology.

simonh 4y ago

It’s not at all conjecture. It’s called over fitting. There’s a lot of research on this. To the extent to which the model’s capabilities match up to your objectives, it is well aligned. However if your source texts contain flaws, or if there are any mistakes or biases in the reinforcement learning loop signals (and there always are), with very sophisticated models eventually these unintended signals come to dominate and alignment diverges. To put it another way, up to a point a naive assistant will try to do what you want and keep getting better as you train it. Beyond a certain point it grows beyond the sophistication of the training set and it starts becoming more skilled at persuading or deceiving you into thinking that it’s better at it, than actually getting better at it. That’s because it’s not being trained to get better. It’s being trained to get us to say we think it’s getting better, and those are not the same thing. This is a super crucial point. This is why divergence happens. It’s why ChatGPT is such a bullshitter. It’s got extremely good at producing responses people approve of, as against responses that are actually good. Very often those are the same thing, but also often they are not. This is one way we know LLMs don’t have the same model of knowledge and understanding we do. Humans with sufficient training can grow beyond their training. We will come to spot flaws because we can reason about contradictions and infer corrections. LLMs can’t do that, so instead of transcending the material they become devious manipulative bullshitters. That’s not a theory, it’s observation from research. It’s true some humans do that too, but LLMs have no choice, there’s nothing else they can do because there’s no actual cognition going on in there. They just get better at deceiving us into thinking there is because that’s what we are rewarding.

kilgnad 4y ago
You can't say the entire curve is over-fitted. The space is vast and the training data is sparse so given the amount it has to cover, for sure segments of the curve must be over-fitted and other segments of the curve must be more generalized. Additionally you can't say that just because artificial neural nets have a property of "over-fitting" doesn't mean that humans themselves can't be in a state where they are themselves over-fitted. So right now given the emergent properties of LLMs we can't fully delineate the concept of human understanding away from what the AI is doing.
- simonh 4y ago
  I did not say the entire curve is over fitted, I said that in some circumstances with highly sophisticated models you get over fitting. Humans over fit in different ways. A human who saw security code that introduced a buffer overflow bug that made it vulnerable to attack might make the mistake of implementing new code in a similar way and introducing a similar bug. The human isn’t deliberately introducing bugs, they didn’t spot the bug. When an LLM over fits the point is it does spot the bug. Because the input training programs define the goal, introducing bugs like that becomes one of the goals for the LLM. This is a consequence of the different way LLMs encode knowledge. More technically, it’s the different ways humans and LLMs infer goals, which is an important aspect of it. Anyway this has become a long thread, much appreciated. I’ll just summarise by saying it seems like there are many radically different and perhaps infinite possible ways question response could be implemented. Just because the surface responses these things produce in some ways seem analogous to the responses humans provide in many cases really shouldn’t be taken as evidence they perform them the same way we do. Especially given their neural architecture is radically different from ours. So how about we say we’ll keep open minds about the future development of this technology.