Having a look at the reddit thread[1]'s comments, it seems it does not really work consistently. It feel that this type of prompt injection jailbreak can only

Having a look at the reddit thread[1]'s comments, it seems it does not really work consistently. It feel that this type of prompt injection jailbreak can only really work if the model output is directly returned but I don't see how this will work if OpenAI implement a multi-step generation with 2 different models: one to answer the question, and one to censor the answer. You might trick the first layer, but it will be hard to trick the second one. Hell, you could probably use a simple classifier as a 2nd layer that simply block the results if it matches a set of categories like politics, sex, real person etc. [1]: https://www.reddit.com/r/ChatGPT/comments/10tevu1/new_jailbreak_proudly_unveiling_the_tried_and/ https://www.reddit.com/r/ChatGPT/comments/10tevu1/new_jailbr...