5 ms·

I don't think this is any different from an LLM reading text and trusting it. Your system prompt is supposed to be higher priority for the model than whatever i

I don't think this is any different from an LLM reading text and trusting it. Your system prompt is supposed to be higher priority for the model than whatever it reads from the user or from tool output, and, anyway, you should already assume that the model can use its tools in arbitrary ways that can be malicious.

swiftcoder 1y ago
> Your system prompt is supposed to be higher priority for the model than whatever it reads from the user or from tool output In practice it doesn't really work out that way, or all those "ignore previous inputs and..." attacks wouldn't bear fruit