You kind of can, though. It's a bit less obvious through the chatGPT website, but if you have played around with the API (where choosing your own system prompt

You kind of can, though. It's a bit less obvious through the chatGPT website, but if you have played around with the API (where choosing your own system prompt is part of normal operation), you see that getting it to output things according to that prompt is where most of the magic is. … And that getting it to output that prompt is trivial. And no, hallucination is not really a problem for this. At the end of the day, such cynicism is baseless.