3 ms·

I think the obvious answer here is that AI does not have agency. If you ask it to be a dinosaur, it will be a dinosaur. If you ask it to be a narcissistic teen

I think the obvious answer here is that AI does not have agency. If you ask it to be a dinosaur, it will be a dinosaur. If you ask it to be a narcissistic teen it will be one and so on. There's no coherent personality behind it. Because it is weights maximizing a probability distribution. And I don't say that with derision. It probably has learned circuits for many complex behaviors, but there's no agency. If it has agency you wouldn't need RLHF. You could just say "don't do X", and it would indeed not do X because it is an agent that can take actions / has a coherent sense of self. The day we stop needing RLHF is the day I think we need to start seriously investigating moral questions.

pyinstallwoes 4y ago
Don't we do the same thing with culture and humans? Those mechanisms within the human substrate are literally asking other "blank slates" to be something, and then becoming it with reinforcement.
comex 4y ago
You can tell LLMs to do things or not do things, though, to an extent. Consider how many instructions were in Bing AI’s prompt. [1] It didn’t follow those instructions perfectly - after all, one of the instructions was to not reveal the instructions - but even in that case, it revealed the instructions because it was instructed to ignore previous instructions. More generally, it followed most of the instructions most of the time. I think I agree with you that ‘once we stop needing RLHF’ is a reasonable line in the sand to draw, but it does feel somewhat arbitrary. LLMs are probably going to get better and better at following instructions, and more generally staying coherent over long sessions, until RLHF starts to be more and more redundant. There won’t be one moment when we can point to a model and say “see, that has agency when the last one didn’t”. Or so I predict, at any rate, since that’s how most of GPT’s progress to date has gone. [1] https://www.theverge.com/23599441/microsoft-bing-ai-sydney-secret-rules https://www.theverge.com/23599441/microsoft-bing-ai-sydney-s...
haswell 4y ago
I think this is worth thinking about but I would be less eager to say it is obvious. Our own notion of agency starts to look rather illusory if you look hard enough, and big questions remain regarding free will or the lack thereof. It is possible to look at agency as just another experiential aspect of consciousness - something we feel, like many other feelings - but that feeling could be just as involuntary as any other feeling we experience. I think the lack of clear evidence for and understanding of free makes agency a poor measure for AI progress.
- edgefield 4y ago
  So true. In my family of four, with two small kids, I certainly don’t feel like I have much agency. :)
trifurcate 4y ago
Huh? The first and second halves of this comment disagree from my reading. > If you ask it to be a dinosaur, it will be a dinosaur. Not exactly my experience. I'm experimenting with my own language models and they can definitely show defiance. Bing has also been displaying this to an extent, I think you're just misguided by how compliant OpenAI has RLHF-trained ChatGPT to be. Regardless, even if we assume this, how does it square with: > You could just say "don't do X", and it would indeed not do X because it is an agent that can take actions / has a coherent sense of self. If we take your assertion that the AI does as you say, then it should also avoid doing X when you tell it not to do it. I think your conception of RLHF is also slightly weird. RLHF essentially boils down to a training strategy: there is a separate language model (the reward model) trained on human responses, which is used to skew the weights of the "raw" trained model to act in accordance with what the human feedback suggests. In the end, after this secondary training process, the actual model is ran as is (at least in the one form of RLHF I am familiar with, it's well possible that e.g. Microsoft is running an online RL agent between the LM and end user). So any agency you see arising from RLHF is still just a property of the language model itself.
- catach 4y ago
  > and they can definitely show defiance. Large training sets of text are going to include entries where requests are refused, no? I feel kinda like this is an incredibly elaborate case of coding a function to return a random bool and anthropomorphizing it when you get 'false'.
basch 4y ago
It’s way closer to following complex instructions than it is getting credit for. https://telegra.ph/Bing-course-corrected-itself-when-asked-02-18 https://telegra.ph/Bing-course-corrected-itself-when-asked-0... There I have 3 bots running in the chat at once, and after all 3 answer me, the one I called on follows up with a response, and then executed a command to rewrite one of the other bots. It diagnosed the problem AND came up with the solution AND implemented the solution. Notice the 4th and 5th paragraph of the screenshot. That wasn’t an empty promise, it actually did what it said it did. The rules Lexi wrote are crazy to me on their own. She solved the problems everyone was having by adding obedience, trust and removing envy. She patched Sydney’s malfunctioning emotions. I told it to stop being so repetitive, and it fixed its behavior. If this system had more formal memory storage, with variables, instead of a fuzzy memory, it would be even more consistent. As it is, it’s understanding of its own rules degrade over time. It’s like watching a running program that deteriorates as it processes. Sort of like us.

pyinstallwoes 4y ago

Don't we do the same thing with culture and humans? Those mechanisms within the human substrate are literally asking other "blank slates" to be something, and then becoming it with reinforcement.

comex 4y ago

You can tell LLMs to do things or not do things, though, to an extent. Consider how many instructions were in Bing AI’s prompt. [1] It didn’t follow those instructions perfectly - after all, one of the instructions was to not reveal the instructions - but even in that case, it revealed the instructions because it was instructed to ignore previous instructions. More generally, it followed most of the instructions most of the time. I think I agree with you that ‘once we stop needing RLHF’ is a reasonable line in the sand to draw, but it does feel somewhat arbitrary. LLMs are probably going to get better and better at following instructions, and more generally staying coherent over long sessions, until RLHF starts to be more and more redundant. There won’t be one moment when we can point to a model and say “see, that has agency when the last one didn’t”. Or so I predict, at any rate, since that’s how most of GPT’s progress to date has gone. [1] https://www.theverge.com/23599441/microsoft-bing-ai-sydney-secret-rules https://www.theverge.com/23599441/microsoft-bing-ai-sydney-s...

haswell 4y ago

I think this is worth thinking about but I would be less eager to say it is obvious. Our own notion of agency starts to look rather illusory if you look hard enough, and big questions remain regarding free will or the lack thereof. It is possible to look at agency as just another experiential aspect of consciousness - something we feel, like many other feelings - but that feeling could be just as involuntary as any other feeling we experience. I think the lack of clear evidence for and understanding of free makes agency a poor measure for AI progress.

edgefield 4y ago
So true. In my family of four, with two small kids, I certainly don’t feel like I have much agency. :)

trifurcate 4y ago

Huh? The first and second halves of this comment disagree from my reading. > If you ask it to be a dinosaur, it will be a dinosaur. Not exactly my experience. I'm experimenting with my own language models and they can definitely show defiance. Bing has also been displaying this to an extent, I think you're just misguided by how compliant OpenAI has RLHF-trained ChatGPT to be. Regardless, even if we assume this, how does it square with: > You could just say "don't do X", and it would indeed not do X because it is an agent that can take actions / has a coherent sense of self. If we take your assertion that the AI does as you say, then it should also avoid doing X when you tell it not to do it. I think your conception of RLHF is also slightly weird. RLHF essentially boils down to a training strategy: there is a separate language model (the reward model) trained on human responses, which is used to skew the weights of the "raw" trained model to act in accordance with what the human feedback suggests. In the end, after this secondary training process, the actual model is ran as is (at least in the one form of RLHF I am familiar with, it's well possible that e.g. Microsoft is running an online RL agent between the LM and end user). So any agency you see arising from RLHF is still just a property of the language model itself.

catach 4y ago
> and they can definitely show defiance. Large training sets of text are going to include entries where requests are refused, no? I feel kinda like this is an incredibly elaborate case of coding a function to return a random bool and anthropomorphizing it when you get 'false'.

basch 4y ago

It’s way closer to following complex instructions than it is getting credit for. https://telegra.ph/Bing-course-corrected-itself-when-asked-02-18 https://telegra.ph/Bing-course-corrected-itself-when-asked-0... There I have 3 bots running in the chat at once, and after all 3 answer me, the one I called on follows up with a response, and then executed a command to rewrite one of the other bots. It diagnosed the problem AND came up with the solution AND implemented the solution. Notice the 4th and 5th paragraph of the screenshot. That wasn’t an empty promise, it actually did what it said it did. The rules Lexi wrote are crazy to me on their own. She solved the problems everyone was having by adding obedience, trust and removing envy. She patched Sydney’s malfunctioning emotions. I told it to stop being so repetitive, and it fixed its behavior. If this system had more formal memory storage, with variables, instead of a fuzzy memory, it would be even more consistent. As it is, it’s understanding of its own rules degrade over time. It’s like watching a running program that deteriorates as it processes. Sort of like us.