3 ms·

Your comment feels like an oversimplification of the post. The post doesn't contend that LLMs are capable of role-playing - that's basically the foundation tha

Your comment feels like an oversimplification of the post. The post doesn't contend that LLMs are capable of role-playing - that's basically the foundation that it builds off of. But saying "LLMs are good at roleplaying" fails to describe why, in the cases the author describes, an LLM can arguably be bad at role-playing. Why does it seem easy to have an LLM switch from following a well-described role to its deceptive opposite, and then often not back the other way? How also do you explain the author's claim that attacking an LLM's pre-imposed prompt with the Waluigi Theory in mind is particularly effective? If an LLM is just good at role-playing, why doesn't it play the role it has already been given by its creator, rather than adapting to the new, conflicting role (including massive rule violations) provided by the user?

PaulHoule 4y ago
A sequence generator is going to flip flop between internal states to fill out a sequence, whether it is a slot that holds the name of an MLB team or a quote (real or imagined) from another document or a quote of a character in a dialogue or a transition from an abstract to the other parts of the paper, etc. If the system is responding to different parts of the prompt it is going to be attending to one part of the prompt when it is outputting something relative to that part of the prompt and attending to another part of the prompt where it is attending to another part of the prompt. There are numerous ways this can go wrong, frequently when somebody gets a chatbot to go rouge they talked with it for a long time, to the point where the beginning of the prompt left the attention window long ago and now it is attending to the text it generated as a result to the prompt and of course the alignment will go bad the same way that you'll make a bunch of wood blocks of irregular sizes if use block N as a template to make block N+1.

PaulHoule 4y ago

A sequence generator is going to flip flop between internal states to fill out a sequence, whether it is a slot that holds the name of an MLB team or a quote (real or imagined) from another document or a quote of a character in a dialogue or a transition from an abstract to the other parts of the paper, etc. If the system is responding to different parts of the prompt it is going to be attending to one part of the prompt when it is outputting something relative to that part of the prompt and attending to another part of the prompt where it is attending to another part of the prompt. There are numerous ways this can go wrong, frequently when somebody gets a chatbot to go rouge they talked with it for a long time, to the point where the beginning of the prompt left the attention window long ago and now it is attending to the text it generated as a result to the prompt and of course the alignment will go bad the same way that you'll make a bunch of wood blocks of irregular sizes if use block N as a template to make block N+1.