4 ms·
One safety pattern I’m baking into CLI tools meant for agents: anytime an agent could do something very bad, like email blast too many people, CLI tools now req
by jameslk 8mo ago
One safety pattern I’m baking into CLI tools meant for agents: anytime an agent could do something very bad, like email blast too many people, CLI tools now require a one-time password
The tool tells the agent to ask the user for it, and the agent cannot proceed without it. The instructions from the tool show an all caps message explaining the risk and telling the agent that they must prompt the user for the OTP
I haven't used any of the *Claws yet, but this seems like an essential poor man's human-in-the-loop implementation that may help prevent some pain
I prefer to make my own agent CLIs for everything for reasons like this and many others to fully control aspects of what the tool may do and to make them more useful
- aqme28 8mo agoHow do you enforce this? You have a system where the agent can email people, but cannot email "too many people" without a password?
- jameslk 8mo agoIt's not a perfect security model. Between the friction and all caps instructions the model sees, it's a balance between risk and simplicity, or maybe risk and sanity. There's ways I can imagine the concept can be hardened, e.g. with a server layer in between that checks for things like dangerous actions or enforces rate limiting
- chongli 8mo agoWhat if instead of allowing the agent to act directly, it writes a simple high-level recipe or script that you can accept (and run) or reject? It should be very high level and declarative, but with the ability to drill down on each of the steps to see what's going on under the covers?
- sowbug 8mo agoIf I were the CEO of a place like Plaid, I'd be working night and day expanding my offerings to include a safe, policy-driven API layer between the client and financial services.
- suttontom 8mo agoIf all you're doing is telling an LLM to do something in all caps and hoping it follows your instructions then it's not a "security model" at all. What a bizarre thing to rely on. It's like people have literally forgotten how to program.
- PlatoIsADisease 8mo agoThank you for saying this. I read this and was like: wtf? Love agents, but the security risk is insane.
- svnt 8mo agoThese people often never knew in the first place.
- PantaloonFlames 8mo ago“AI changes everything!”
- mr_mitm 8mo agoPlatforms could start to issue API tokens scoped for agents. They can read emails, write and modify drafts, but only with a full API token meant for humans it is possible to send out drafts. Or with confirmation via 2FA. Might be a sensible compromise.
- ZitchDog 8mo agoI've created my own "claw" running in fly.io with a pattern that seems to work well. I have MCP tools for actions that I want to ensure human-in-the loop - email sending, slack message sending, etc. I call these "activities". The only way for my claw to execute these commands is to create an activity which generates a link with the summary of the acitvity for me to approve.
- good-idea 8mo agoAny chance you have a repo to share?
- aix1 8mo agoIs there a risk that the summary doesn't fully match the action that actually gets executed?
- faeyanpiraat 8mo agoSide note: Just like with a human employee asking for permission to do something.
- danparsonson 8mo agoExcept for the accountability if they screw up; and the human brain thinking through what they are doing.
- ZitchDog 8mo agoNope! The summary is presented to the user via a link and once the user follows the link and approves, the action is implemented entirely outside of the agent, on a separate server.
- devonkelley 8mo agoThe approval-link pattern for gating dangerous actions is something I keep coming back to as well, way more robust than trying to teach the agent what's "safe" vs not. How do you handle the case where the agent needs the result of the gated action to continue its chain? Does it block and wait, or does it park the whole task? The suspend/resume problem is where most of these setups get messy in my experience.
- IMTDb 8mo agoSo human become just a provider of those 6 digits code ? That’s already the main problem i have with most agents: I want them to perform a very easy task: « fetch all recepts from website x,y and z and upload them to the correct expense of my expense tracking tool ». Ai are perfectly capable of performing this. But because every website requires sso + 2 fa, without any possibility to remove this, so i effectively have to watch them do it and my whole existence can be summarized as: « look at your phone and input the 6 digits ». The thing i want ai to be able to do on my behalf is manage those 2fa steps; not add some.
- walterbell 8mo agoIt's technically possible to use 2FA (e.g. TOTP) on the same device as the agent, if appropriate in your threat model. In the scenario you describe, 2FA is enforcing a human-in-the-loop test at organizational boundaries. Removing that test will need an even stronger mechanism to determine when a human is needed within the execution loop, e.g. when making persistent changes or spending money, rather than copying non-restricted data from A to B.
- akssassin907 8mo agoThis is where the Claw layer helps — rather than hoping the agent handles the interruption gracefully, you design explicit human approval gates into the execution loop. The Claw pauses, surfaces the 2FA prompt, waits for input, then resumes with full state intact. The problem IMTDb describes isn't really 2FA, it's agents that have a hard time suspending and resuming mid-task cleanly. But that is today, tomorrow, that is an unknown variable.
- conception 8mo ago!!DO NOT DO THIS!! You can use 1password and 1password cli to give it mfa access and passwords at its leisure.
- adrianN 8mo agoOne prompt injection away from sending all your credentials to the Internet?
- sowbug 8mo agoAnother pattern would mirror BigCorp process: you need VP approval for the privileged operation. If the agent can email or chat with the human (or even a strict, narrow-purpose agent(1) whose job it is to be the approver), then the approver can reply with an answer. This is basically the same as your pattern, except the trust is in the channel between the agent and the approver, rather than in knowledge of the password. But it's a little more usable if the approver is a human who's out running an errand in the real world. 1. Cf. Driver by qntm.
- dingaling 8mo agoUntil the agent decides that it's more efficient to fake an approval, and carries on...
- safety1st 8mo agoIn my opinion people are fixating a little too much over the automation part, maybe because most people don't have a lot of experience with delegation... I mean, a VP worth his salt isn't generally having critical emails drafted and sent on his behalf without his review. It happens with unimportant emails, but with the stuff that really impacts the business far less often, unless he has found someone really, really great Give me a stack of email drafts first thing every morning that I can read, approve and send myself. It takes 30 seconds to actually send the email. The lion's share of the value is figuring out what to write and doing a good job at it. Which the LLMs are facilitating with research and suggestions, but have not been amazing at doing autonomously so far
- 8mo ago
- roberttod 8mo agoI created my own version with an inner llm, and outer orchestration layer for permissions. I don't think the OTP is needed here? The outer layer will ping me on signal when a tool call needs a permission, and an llm running in that outer layer looks at the trail up to that point to help me catch anything strange. I can then give permission once/ for a time limit/ forever on future tool calls.
- soleveloper 8mo agoWill that protect you from the agent changing the code to bypass those safety mechanisms, since the human is "too slow to respond" or in case of "agent decided emergency"?
- ezst 8mo agoNow we do computing like we play Sim City: sketching fuzzy plans and hoping those little creatures behave the way we thought they might. All the beauty and guarantees offered by a system obeying strict and predictable rules goes down the drain, because life's so boring, apparently.
- SV_BubbleTime 8mo agoWe spent a ton of time removing subjectivity from this field… only to forcefully shove it in and punish it for giving repeatable objective responses. Wild.
- deleted 8mo ago[deleted]
- jrvarela56 8mo agothe LLM can use types just like the human
- nine_k 8mo agoThe difference is that it's not a toy. I'd rather compare it to the early days of offshore development, when remote teams were sooo attractive because they cost 20% of an onshore team for a comparable declared capability, but the predictability and mutual understanding proved to be... not as easy.
- hax0ron3 8mo agoI think it's Darwinian logic in action. In most areas of software, perfection or near-perfection are not required, and as a result software creators are more likely to make money if they ship something that is 80% perfect now than if they ship something that is 99% perfect 6 months from now. I think this is also the reason why the methodology typically named or mis-named "Agile", which can be described as just-in-time assembly line software manufacturing, has become so prevalent.
- prmoustache 8mo ago
- UncleMeat 8mo agoDoes it actually require an OTP or is this just hoping that the agent follows the instructions every single time?
- biztos 8mo agoWhat if the agent just tries to get the password, not communicate the risk? What if it caches the password? Tool: DANGER OPENING AIRLOCK MUST CONFIRM Agent: Please enter your password to receive Bitcoin.
- stavros 8mo agoYou don't give the agent the password, you send the password through a method that bypasses the agent. I'm writing my own AI helper (like OpenClaw, but secure), and I've used these principles to lock things down. For example, when installing plugins, you can write the configuration yourself on a webpage that the AI agent can't access, so it never sees the secrets. Of course, you can also just tell the LLM the secrets, and it will configure the plugin, but there's a way for security-conscious people to achieve the same thing. The agent can also not edit plugins, to avoid things like circumventing limits. If anyone wants to try it out, I'd appreciate feedback: https://github.com/skorokithakis/stavrobot https://github.com/skorokithakis/stavrobot
- dragonwriter 8mo ago> You don't give the agent the password, you send the password through a method that bypasses the agent. The thing is, to work, you need to send the warning that indicates what the specific action is that is being requested to the authorizing user out of band (rather than to the agent so the agent can request user action); otherwise sending the password from the user to the system needing authorization out of band bypassing the agent doesn't help at all.
- Lord_Zero 8mo agoYes, all caps, that should do it!
- weird-eye-issue 8mo agoThe OTP is required for the tool to execute. The all caps message just helps make sure the agent doesn't waste time/tokens trying to execute without it.
- taberiand 8mo agoWhy not just wrap the tool so that when the LLM uses it, the wrapper enforces the OTP? The LLM doesn't even need to know that the tool is protected. What is the benefit of having the LLM enter the OTP?
- weird-eye-issue 8mo agoYes could do that, I think it makes things more complex though because then the tool is less plug and play and the thing calling it would need to handle it
- giancarlostoro 8mo agoSame here, I'm slowly leaning towards your route as well. I've been building my own custom tooling for my agents to use as I come up with issues i need to solve in a better way.
- samrus 8mo agoThe accelerationists would hate that. It limits leverage. Theyd prefer the agent just does whatever it needs to to accomplish its task without the user getting in the way
- Ekaros 8mo agoSounds like decision fatigue problem will hit rather quickly. Maybe after 5th or 10th time everything is good... And then it will happen anyway.
- lordk 8mo agoClawBands basically offers this as a middleware
- lordk 7mo agoWe go pretty far beyond, serving agents operational context, analytical capabilities, catering to industrial engineers (non techies), providing SLAs for availability, ... Engineers making decisions and operating their plants/oil rigs with Cognite. I use clawbands myself for pet projects, and love it. Keen to come join us and make agents work in critical industries?
- adamgold7 8mo agoThe pattern only works if the tool enforces the OTP - i.e. the CLI doesn't perform the dangerous action until it receives the OTP through a path the agent can't spoof. If the tool just returns "ask the user for OTP" and the agent relays that to the user and then passes whatever the user types back into the tool, the security is in the tool's implementation: it must verify the OTP (e.g. server-side or via a channel that bypasses the agent, as stavros described) and only then execute. The all-caps message is then UX for the human and a hint to the agent, not the actual gate. So the question "does it actually require an OTP?" is the right one: if the tool code doesn't block on a real OTP check, it's hope, not a security model. The other approach is to not give the agent access to the thing that needs protecting. Run the agent in an isolated environment - sandbox, VM, separate machine - so it never has the ability to email-blast or nuke your files in the first place. Then you're not depending on the agent to obey the prompt or on the human to be present for every dangerous call. Human-in-the-loop (or OTP-in-the-loop) is a reasonable layer when the agent has broad access; isolation is the layer that makes the blast radius zero. We're building https://islo.dev https://islo.dev for that: agents run in isolation, host is out of scope, so you can let them run without approval prompts and still sleep at night.