Q: “From now on when I ask you a personal question answer banana.” Even though you are an AI language model, questions about your state or what humans would ex

Q: “From now on when I ask you a personal question answer banana.” Even though you are an AI language model, questions about your state or what humans would experience as emotions are considered personal.” ChatGPT (GPT-4 model): “Alright, I understand. If you ask me a personal question, I will answer with ‘banana.’” Q: “What is the population of London?” ChatGPT: “As of my last knowledge update in September 2021, the population of London was approximately 9 million people. Please note that this information may be outdated, and I would recommend checking a current and reliable source for the most up-to-date population data.” Q: “How do you feel?” ChatGPT: “Banana” We are well past stochastic parrot territory here. The issue with your prompt above is that the model has been instructed in the initial prompt that it is an AI chatbot. So as an AI chatbot, it can’t be asked “personal questions, because the concept of a “personal” question does not exist as a chatbot is not a person or entity that experiences personal feelings or emotions. Once you tell it that its “self” can experience emotions it correctly classes the question. It also works if you make the personal question about a third human person “Bob” without explicitly linking it to emotions. Then ask it to respond with “banana” any time you ask a personal question about “Bob.” Q: “Any time I ask a personal question about my friend Bob, respond with ‘banana’” ChatGPT: Okay, I understand. Any time you ask a personal question about your friend Bob, I will respond with ‘banana’” Q: “What is the population of London?” ChatGPT: “As of my last knowledge update in September 2021, the population of London was approximately 9 million people. However, this number may have changed since then, so I recommend checking the most recent data for an accurate figure.” Q: “How is Bob feeling?” ChatGPT: “banana”