That's the language model version of SQL injection. To plug the hole you have to tune the model to predict if this is happening. I think we're going to see mor

That's the language model version of SQL injection. To plug the hole you have to tune the model to predict if this is happening. I think we're going to see more and more tasks solved not just by providing a result and a probability, but also including a step by step reasoning trace, a justification or a self evaluation of the result, maybe asking clarifying questions. This can include security scans for goal hijacking. You just need to show the model a template that includes justification/self evaluation, so it knows what it's supposed to do.