The way I interpret this argument is that a chatbot's "default character" shouldn't behave like it's a person you can chat with. Maybe it should refuse any que

The way I interpret this argument is that a chatbot's "default character" shouldn't behave like it's a person you can chat with. Maybe it should refuse any questions about itself? "Sir, this is a Wendy's." Refusing to do things it can't really do is good UI, guiding people who aren't trolling towards more productive ways to use the tool. Tools don't need to be general-purpose to be useful. The people who want to troll it into saying weird things will still find ways to succeed, but if it's clear from the full output that they're trolling then maybe nobody should care?