3 ms·
Except agents actually have an intent, and can route around obstacles to accomplish that intent. If you merely block a specific action, they will find another
by Legend2440 6mo ago
Except agents actually have an intent, and can route around obstacles to accomplish that intent.
If you merely block a specific action, they will find another way to do what they're trying to do. Agent security requires controlling the agent's intent.
- sigbottle 6mo agoThe goal behind most "clean" software design in general is to eliminate the possibility of failure via constraints. That's the pattern I've seen over the years. Of course, the map is not the territory - you need to make sure the reachable set within the constraints is actually a subset of the real reachable set. Which may be underspecified or unknown a priori (as if you could've really specified the true reachable set, why didn't you just encode those rules?) So I'm sympathetic to the criticism, especially since composition of formal methods & analyzing their effects is still very much a hard problem (and not just computationally - philosophically, often, for the reason I listed above). That being said, I don't know a better solution. Begging the agent with prompts doesn't work. Are you suggesting some kind of mechanistic interpretability, maybe?
- deleted 6mo ago[deleted]