4 ms·
It's jarring because you can give it an instruction like "I want you to talk like a pirate from now on" or "Please end every reply with 'beep boop'" and it will
by sltkr 3y ago
It's jarring because you can give it an instruction like "I want you to talk like a pirate from now on" or "Please end every reply with 'beep boop'" and it will happily oblige and continue to do what was asked continually. So it's strange that being apologetic is apparently so ingrained in its training that it cannot stop apologizing even when you specifically requested that it refrained from doing so.
- namaria 3y agoI think it's more strange that the coincidence of some imperative prompts generating texts that looks like it's answering to it makes intelligent people think it's reasoning about the instructions and responding meaningfully to them instead of, you know, generating text as it was programmed to do.
- HarHarVeryFunny 3y agoIt treats the system prompt specially - it's more than just a prefix to the conversation. OpenAI have said they put a lot of work it making it follow it as closely as possible (but no details of how they did it).