It's problematic for the same reason other injection attacks are. It allows an attacker to get their "code" running in what should be a privileged environment y

It's problematic for the same reason other injection attacks are. It allows an attacker to get their "code" running in what should be a privileged environment you control. In this case, the "code" is a natural language prompt, but the principle applies. Think about what would happen if an app scraped a web page on which someone had placed instructions for an LLM. The app then passes the instructions as context to the LLM, which is using the ReAct model [0] so it can run code, communicate with other apps, download content from the web, and so on. It's dangerous even if the LLM is just instructed to output JSON which the app uses for some other purpose. Prompt injection is a fairly trivial problem if you're using an LLM to write a blog post or research something. But the deeper LLMs are integrated into complex applications, the more dangerous it becomes. [0]: https://arxiv.org/abs/2210.03629 https://arxiv.org/abs/2210.03629