There indeed is a strong overlap with social engineering, but in my view the whole reason why social engineering the LLM is possible is an "injection vulnerabil

There indeed is a strong overlap with social engineering, but in my view the whole reason why social engineering the LLM is possible is an "injection vulnerability". We don't want the LLM to treat third-party data in the same way as the communication with the user. We want the user to be able to talk with an LLM-based chatbot in arbitrary ways and issue arbitrary instructions, however, we also want a strict separation between these instructions and the data they operate on, so that when the user says "fix style problems in that blob of text" the model has the capability to tell that this blob of text is fundamentally different from the instructions, and that literally nothing in it should even theoretically enable social engineering.