4 ms·
The fact that the words "structured" or "constrained" generation continue not to be uttered as the beginning of how you mitigate or solve this shows just how fe
by Der_Einzige 1y ago
The fact that the words "structured" or "constrained" generation continue not to be uttered as the beginning of how you mitigate or solve this shows just how few people actually build AI agents.
- dragonwriter 1y agoStructured/constrained generation doesn't protect against outside prompt injection, or protect against the prompt injection causing incorrect use of any facility the system is empowered to use. It can narrow the attack surface for a prompt injection against one stage of an agentic system producing a prompt injection by that stage against another stage of the system, but it doesn’t protect against a prompt injection producing a wrong-but-valid output from the stage where it is directly encountered, producing a cascade of undesired behavior in the system.
- roywiggins 1y agoBest you can do is constrain responses to follow a schema, but if that schema has any free text you can still poison the context, surely? Like if I instruct an agent to read an email and take an appropriate action, and the email has a prompt injection that tells it to take a bad action instead of a good action, I am not sure how structured generation helps mitigate the issue at all.