8 ms·

Designing agentic loops

284 points by simonw 1y ago

simonw 1y ago
I updated this post to link to the Claude Code docs that suggest running YOLO mode using their Docker dev container: https://www.anthropic.com/engineering/claude-code-best-practices#d-safe-yolo-mode https://www.anthropic.com/engineering/claude-code-best-pract... - which locks down network access to just a small set of domains: https://github.com/anthropics/claude-code/blob/5062ed93fc67f9322f807ecbf391ae4376cf8e83/.devcontainer/init-firewall.sh#L67-L76 https://github.com/anthropics/claude-code/blob/5062ed93fc67f...
saltyoldman 1y ago
I wouldn't be surprised if agents start getting managed by a distributed agentic system - think about it. Right now you get codex/claude/etc... and it's system prompt and various other internally managed prompts are locked down to the version you downloaded. What if a distributed system ran experimental prompts and monitored the success rate (what code makes it into a commit) and provides feedback to the agent manager. That could help automatically fine tune it's own prompts.
- datadrivenangel 1y ago
  This is what Anthropic does for their "High Compute" SWE benchmarking: " For our "high compute" numbers we adopt additional complexity and parallel test-time compute as follows: - We sample multiple parallel attempts. - We discard patches that break the visible regression tests in the repository, similar to the rejection sampling approach adopted by Agentless (Xia et al. 2024); note no hidden test information is used. - We then use an internal scoring model to select the best candidate from the remaining attempts. - This results in a score of 82.0% for Sonnet 4.5. " But running in parallel means you use more compute, meaning that the cost is higher. Good results are worth paying for, but if a super recursive approach costs more and takes longer than a human...
mccoyb 1y ago
I think this is a strictly worse name than "agentic harness", which is already a term used by open-source agentic IDEs (https://github.com/search?q=repo%3Aopenai%2Fcodex%20harness&type=code https://github.com/search?q=repo%3Aopenai%2Fcodex%20harness&... or https://github.com/openai/codex/discussions/1174 https://github.com/openai/codex/discussions/1174) Any reason why you want to rename it? Edit: to say more about my opinions, "agentic loop" could mean a few things -- it could mean the thing you say, or it could mean calling multiple individual agents in a loop ... whereas "agentic harness" evokes a sort of interface between the LLM and the digital outside world which mediates how the LLM embodies itself in that world. That latter thing is exactly what you're describing, as far as I can tell.
- simonw 1y ago
  I like "agentic harness" too, but that's not the name of a skill. "Designing agentic loops" describes a skill people need to develop. "Designing agentic harnesses" sounds more to me like you're designing a tool like Claude Code from scratch. Plus "designing agentic loops" includes a reference to my preferred definition of the term "agent" itself - a thing that runs tools in a loop to achieve a goal.
  - prats226 1y ago
    Context engineering is another name people have given to same skill?
    - simonw 1y ago
      I think that's actually quite different. Context engineering is about making sure you've stuffed the context with all of the necessary information - relevant library documentation and examples and suchlike. Design the agentic loop is about picking the right tools to be provided to the model. The tool descriptions may go in the context but you also need to provide the right implementations of them.
      - tptacek 1y ago
deleted 1y ago
[deleted]
mike_hearn 1y ago
I recently built my own coding agent, due to dissatisfaction with the ones that are out there (though the Claude Code UI is very nice). It works as suggested in the article. It starts a custom Docker container and asks the model, GPT-5 in this case, to send shell scripts down the wire which are then run in the container. The container is augmented with some extra CLI tools to make the agent's life easier. My agent has a few other tricks up its sleeve and it's very new, so I'm still experimenting with lots of different ideas, but there are a few things I noticed. One is that GPT-5 is extremely willing to speculate. This is partly because of how I prompt it, but it's willing to write scripts that try five or six things at once in a single script, including things like reading files that might not exist. This level of speculative execution speeds things up dramatically especially as GPT-5 is otherwise a very slow model that likes to think about things a lot. Another is that you can give it very complex "missions" and it will drive things to completion using tactics that I've not seen from other agents. For example, if it needs to check something that's buried in a library dependency, it'll just clone the upstream repository into its home directory and explore that to find what it needs before going back to working on the user's project. None of this triggers any user interaction due to running in the container. In fact, no user interaction is possible. You set it going and do something else until it finishes. The model is very much to queue up "missions" that can then run in parallel and you merge them together at the end. The agent also has a mode where it takes the mission, writes a spec, reviews the spec, updates the spec given the review, codes, reviews the code, etc. Even though it's early days I've set this agent missions that it spent 20 minutes of continuous uninterrupted inferencing time on, and succeeded excellently. I think this UI paradigm is the way to go. You can't scale up AI assisted coding if you're constantly needing to interact with the agent. Getting the most out of models requires maximally exploiting parallelism, so sandboxing is a must.
- simonw 1y ago
  That's really interesting. I've noticed something similar - I've tried frontend tasks against GPT-5-Codex and seen it guess the URL of the underlying library (on jsdelivr or GitHub) and attempt to fetch the original source code, often trying several different URLs, in order to dig through the source and figure out how to use an undocumented API feature.
akashakya 1y ago
For lightweight sandboxing on Linux you can use bubblewrap or firejail instead of Docker. They are faster and _simpler_. Here is a bwrap script I wrote to run Claude in a minimal sandbox an hour back: exec bwrap \ --ro-bind /usr /usr \ --ro-bind /etc /etc \ --ro-bind /run /run \ --ro-bind "$NODE_PATH" /node \ --proc /proc \ --dev /dev \ --symlink usr/lib64 /lib64 \ --tmpfs /tmp \ --unshare-all \ --share-net \ --die-with-parent \ --new-session \ --bind "$HOME/claude" /claude \ --bind "$HOME/.claude.json" /claude/.claude.json \ --bind "$HOME/.claude" /claude/.claude \ --setenv HOME /claude \ --setenv PATH "/node:/claude/bin:/usr/bin" \ --bind "$(pwd)" /work \ --chdir /work \ /claude/bin/claude "$@"
- simonw 1y ago
  Nice, thanks for sharing. The lack of an equivalent on macOS (sandbox-exec is similar but mostly undocumented and described as "deprecated" by Apple) is really frustrating.
  - DowsingSpoon 1y ago
    I had been planning to explore Lima tonight as a mechanism to shackle CC on macOS. The trouble with sandbox-exec is that it’s control over network access is not fine grain enough, and I found its file system controls insufficient. Also, I recently had some bad experiences which lead me to believe the tool MUST be run with strict CPU and memory resource limits, which is tricky on macOS.
    - simonw 1y ago
      Neat, I've not tried https://github.com/lima-vm/lima https://github.com/lima-vm/lima
    - big_toast 1y ago
      Wait, does lima do isolation in a macos context too? It looks like linux vms, which apple's container-cli (among others) covers at a basic level. I'd like apple to start providing macOS images that weren't the whole OS.. unless sandbox-exec/libsandbox have affordance for something close enough? You can basically ask claude/chatgpt to write its jail (dockerfile) and then run that via `container` without installing anything on macos outside the container it builds (IIRC). Even the container-cli will use a container to build your container..
tra3 1y ago
On one hand, folks are complaining that even basic tasks are impossible with LLMs. On the other hand we have cursed language [0] which was fully driven by AI and seems to be functional. Btw, how much did that cost? I feel like I've been hugely successful tools like Claude Code, aider, open code. Especially when I can define custom tools. "You" have to be a part of the loop, in some capacity. To provide guidance and/or direction. I'm puzzled by the fact that people are surprised by this. When I'm working with other entities (people) who are not artificially intelligent, the majority of the time is spent clarifying requirements and aligning on goals. Why would it be different with LLMs? 0: https://ghuntley.com/cursed/ https://ghuntley.com/cursed/
- simonw 1y ago
  The cursed cost estimate was $14,000 https://twitter.com/GeoffreyHuntley/status/1965295152962097550 https://twitter.com/GeoffreyHuntley/status/19652951529620975...
- rv64imafdc 1y ago
  > Btw, how much did that cost? $30k+ Not sure how functional though!
tptacek 1y ago
This is great. I feel like most of the oxygen is going to go to the sandboxing question (fair enough). But I'm kind of obsessed with what agent loops for engineering tasks that aren't coding look like, and also the tweaks you need for agent loops that handle large amounts of anything (source code lines, raw metrics or oTel span data, whatever). There was an interval where the notion of "context engineering" came into fashion, and we quickly dunked all over it (I don't blame anybody for that; "prompt engineering" seemed pretty cringe-y to me), but there's definitely something to the engineering problems of managing a fixed-size context window while iterating indefinitely through a complex problem, and there's all sorts of tricks for handling it.
- kridsdale1 1y ago
  I use agentic loops for document and spreadsheet (aka bullshit to interface with non engineers in the company) since my model of choice is adept at markdown and csv, and the office software can import and export those.
- m4r71n 1y ago
  I can imagine an agentic loop that updates dependencies à la Dependabot/Renovate-style by going through the changelog of a new version, reviewing new code changes, and evaluating whether it's worth it to upgrade (or even dangerous to do so, either from stability or security point of view). Too often these tools are used to blindly respin builds with the latest and greatest versions, which is what gets most people in trouble when their NPM deps become malicious.
  - tptacek 1y ago
    You should write it! You might be surprised how easy it is to get something working.
    - m4r71n 1y ago
      Ha, well look at that, not even a day later: https://fossa.com/blog/fossabot-dependency-upgrade-ai-agent/ https://fossa.com/blog/fossabot-dependency-upgrade-ai-agent/
CuriouslyC 1y ago
One important issue with agentic loops is that agents are lazy, so you need some sort of retrigger mechanism. Claude code supports hooks, you can wire your agent stop hook to a local LLM, feed the context in and ask the model to prompt Claude to continue if needed. It works pretty well, Claude can override retriggers if it's REALLY sure it's done. Regarding sandboxing, VMs are the way. Prompt injected agents WILL be able to escape containers 100%.
- tptacek 1y ago
  Seems "big if true" given the number of cloud providers that use containers for customer isolation.
  - CuriouslyC 1y ago
    This is not speculative, it's happened plenty already. People put mitigations in place, patch libraries and move on. The difference is that agents will find new zero days you've never heard of for stuff on your system people haven't scrutinized adequately. There will be zero advanced notice, and unlike human attackers who need to lie low until they can plan an exit, it'll be able to exploit you heavily right away. Do not take the security impact of agents lightly!
    - tptacek 1y ago
      I feel like my bona fides on this topic are pretty solid (without getting into my background on container vs. VM vs. runtime isolation) and: "the agents will find new zero days" also seems "big if true". I point `claude` at a shell inside a container and tell it "go find a zero day that breaks me out of this container", and you think I'm going to succeed at that? I had assumed you were saying something more like "any attacker that prompt-injects you probably has a container escape in their back pocket they'll just stage through the prompt injection vector", but you apparently meant something way further out.
      - CuriouslyC 1y ago
jujugoboom 1y ago
Wrong takeaway, but claude code was released February of this year??? I swear people have been glazing it for way longer... my memory isn't that bad right?
- simonw 1y ago
  February 24th, Anthropic tucked it into the same announcement as Sonnet 3.7: https://www.anthropic.com/news/claude-3-7-sonnet https://www.anthropic.com/news/claude-3-7-sonnet
refset 1y ago
For anyone else curious about what a practical loop implementation might look like, Steve Yegge YOLO-bootstrapped his 'Efrit' project using a few lines of Elisp: https://github.com/steveyegge/efrit/blob/4feb67574a330cc789fece2d84aeb5e2964f3537/archived/efrit-agent.el#L495-L536 https://github.com/steveyegge/efrit/blob/4feb67574a330cc789f... And for more context on Efrit this is a fun watch: "When Steve Gives Claude Full Access To 50 Years of Emacs Capabilities" https://www.youtube.com/watch?v=ZJUyVVFOXOc https://www.youtube.com/watch?v=ZJUyVVFOXOc
ademup 1y ago
I'm surprised to see so many people using containers when setting up a KVM is so easy, gives the most robust environment possible, and to my knowledge much has better isolation. A vanilla build of Linux plus your IDE of choice and you're off to the races.
- scubbo 1y ago
  Not everyone has spare hardware lying around!
  - ademup 1y ago
    For sure! But just for reference, I'm on a mid-tier 2022 Dell Inspiron laptop: Ryzen 7 5825U with 64GB ram and 500GB SSD. On it, I run Ubuntu 24.04 as my host, and my guest is Lubuntu with 16GB ram and 80GB ssd for my KVM. I almost always have 2 instances of PHPstorm open in both Host and and Guest with multiple terminal tabs running various agentic tasks.
    - tecoholic 1y ago
      64 GB. Say no more :) I lived with 16GB until last year and upgraded to 32 only this year, which I thought was a huge improvement. I suspect a lot of people are around this ballpark, especially if they have bought Macs. Mine is Linux, still. So containers are the “simpler” versions.
    - scubbo 1y ago
      ...wait, you and I are using "KVM" in different ways, then. To me, it means a switch that lets you use the same Keyboard, Monitor ("Video"), and Mouse for two different machines. Sounds like you're talking instead about a technique for running a VM on a single machine - which, from Googlin', I suspect is "Kernel-based Virtual Machine", a new-to-me term. Thanks for teaching me something!
- mike_hearn 1y ago
  You often don't need strong isolation. The sandboxing is more to avoid model accidents than a Skynet scenario.
BoppreH 1y ago
> If anything goes wrong it’s a Microsoft Azure machine somewhere that’s burning CPU and the worst that can happen is code you checked out into the environment might be exfiltrated by an attacker, or bad code might be pushed to the attached GitHub repository. Isn't that risking getting banned from Azure? The compromised agent might not accomplish anything useful, but its attempts might get (correctly!) flagged by the cloud provider.
- simonw 1y ago
  My guess is that most cloud providers have procedures in place to help avoid banning legitimate customers because one of their instances got infected with malware (effectively what a rogue agent would be).
ggggggdgdhdjdj 1y ago
I find the word agentic to be a hot spike into my brain ... I am not sure why but I think it has to do with the fact it is marketing bs. Is it just me ?
- handfuloflight 1y ago
  It's marketing BS in the hands of grifters.
  - simonw 1y ago
    Can you expand on why you think that?
    - handfuloflight 1y ago
      I'm saying there are grifters who co-opt the terms used by people actually pioneering / exploring the space, to sell BS; and that taints OP's view. This doesn't need stating but if it does I am not saying you are a grifter, obviously, because you substantiate your positions. Just another classic signal v. noise.
      - simonw 1y ago
        That makes sense, thanks.
  - nobodywillobsrv 1y ago
    "Agentic" feels like the new "Generative". Yes, it means something "vague" and vibes oriented. It's when you can run things in a terminal or async or something maybe.
    - simonw 1y ago
      I've been trying to make it less vague. I'm using it to mean something that's directly related to running tools in a loop to achieve a goal.
moduspol 1y ago
A little off-topic but I’ve been waiting for a post by someone who’s built something like Claude Code or Codex that can explain how the state machine works for execution. Clearly it’s not just a good system prompt and hoping for the best. Obviously they’ve both got fairly good algorithms for working toward whatever the user asked for incrementally, handling expectable errors, changing course if necessary, etc. I mean ultimately it’s gotta be a really big flow chart that results in the LLM getting prompted various ways in all kinds of directions before ultimately converging on the final response, right? Or am I way off-base? Has anyone seen such a post explaining this architecture / state machine / algorithm?
- zenburnmyface 1y ago
  This is in that direction: https://jannesklaas.github.io/ai/2025/07/20/claude-code-agent-design.html https://jannesklaas.github.io/ai/2025/07/20/claude-code-agen...
  - simonw 1y ago
    (This is a much better answer than my Codex Cloud generated version in the sibling comment.)
  - moduspol 1y ago
    Thanks. Looks like exactly what I was looking for.
- simonw 1y ago
  Since Codex CLI is open source I got Codex Cloud to attempt to answer your question - here's the shared session: https://chatgpt.com/s/cd_68dc99fe3e948191a8923ddc4a1f4310 https://chatgpt.com/s/cd_68dc99fe3e948191a8923ddc4a1f4310 And here's the markdown file it produced with an attempt at an answer: https://github.com/simonw/codex-scratchpad/blob/codex/analyze-codex-source-code-architecture/answer.md https://github.com/simonw/codex-scratchpad/blob/codex/analyz...
- mike_hearn 1y ago
  You'd be surprised. You don't need a state machine to get started, only if you want to get really advanced. The models are smart. It's enough to just ask them to send you instructions, execute them and send back the results.
codazoda 1y ago
I made a simple docker container to jail an LLM for a recent project. https://github.com/codazoda/llm-jail https://github.com/codazoda/llm-jail
cadamsdotcom 1y ago
My preferred ergonomics for agents are like using a washing machine: choose what clothes you want to have cleaned (whites or colors..) and press “go”. Claude Code plan mode is the washing machine of agentic loops. It lets you see a detailed plan and if you approve you can let it rip with trust that it’ll do what it says, and go do something else while it’s at work. Adding tests and other ways to validate its work help it go further & run longer off a single plan.
chevman 1y ago
I feel like the AI industry is basically recreating "workflow engines" again and again and again, in different forms, with different input mechanisms.
- jarboot 1y ago
  Temporal.io has been good for orchestrating agentic workflows, or workflows in general
amluto 1y ago
The state of YOLO is embarrassing: > Update: It turns out Anthropic have their own documentation on Safe YOLO mode for Claude Code which says: > Letting Claude run arbitrary commands is risky and can result in data loss, system corruption, or even data exfiltration (e.g., via prompt injection attacks). To minimize these risks, use --dangerously-skip-permissions in a container without internet access. You can follow this reference implementation using Docker Dev Containers. [https://github.com/anthropics/claude-code/tree/main/.devcontainer https://github.com/anthropics/claude-code/tree/main/.devcont...] And… that link goes to a devcontainer that firewalls itself from inside using, effectively, sudo iptables. If Claude Code can’t break that all by itself in a single try, I’d be a bit surprised.
SafeDusk 1y ago
Resonating with the statement "Designing agentic loops is a very new skill—Claude Code was first released in just February 2025!" when I was investigating Codex[0]. [0]: https://blog.toolkami.com/openai-codex-tools/#coding-agents https://blog.toolkami.com/openai-codex-tools/#coding-agents
lobochrome 1y ago
Is there a way to let the agents share their agents.md? Claude code looks at Claude.md and I have yet to find a way to unify the agent handbook.
- simonw 1y ago
  At this point effectively everyone else is on https://agents.md/ https://agents.md/ - Claude are the only holdout.
  - data-ottawa 1y ago
    Is there any reason this is better than just reading your regular readme or docs? All of these things look helpful for humans too!
    - threecheese 1y ago
      You could say that a Readme is declarative, while an agent file is imperative.
    - simonw 1y ago
      The main difference between AGENTS.md and a README.md is that coding agents will automatically ingest AGENTS.md when they start. You may not want them to ingest README.md if it's long and contains irrelevant information as that might be a waste of valuable tokens.
- mike_hearn 1y ago
  I just use symlinks.
Escapade5160 1y ago
I typically run my code agents in distrobox.
athrowaway3z 1y ago
On unix you can just use a user account to limit access. Create a group for you and the `claude` user, and have it do `umask 002` on login. The shellagent use case has much more parallels with 1980s TTY+mainframes dealing with multiple users than it has with containers. I.e. Docker's USP was reproducibility; not access controls.
mattacular 1y ago
> This is so dangerous, but it’s also key to getting the most productive results where do i sign up?
dboreham 1y ago
In case anyone else is confused: I saw the word "loop" and wondered what the point is to run an LLM in a loop? Like, what's the goal here? You want to write N different React todo web apps, or something? After reading the article a bit, I realized this isn't about "loops" at all. What it is about is "bypassing the yes/no human check on each step the LLM wants to take in completing some multi-step task". You can already to this with, for example, claude code. However, one of the points of having a human approve LLM actions is to prevent it doing something stupid or destructive. So now you have a second problem: how to prevent the LLM from poking holes in spacetime? Answer: sandbox it. And that's what the article is about.
deevus 1y ago
Fantastic article. I'm always looking for better ways to work with agents. I am personally using Claude Code, and want to try some more YOLO type stuff. My problem is I often give it too much to do. I'm usually overly optimistic about how effective it can be. I have found Sonnet 4.5 to be quite the improvement over Sonnet 4 however.
deleted 1y ago
[deleted]
codethief 1y ago
For sandboxing and parallelizing agents people at my day job recently started using container-use[0] and seem to be quite happy with it. I have yet to try it in depth but the approach it takes seems quite sensible to me: Have the agent execute all commands within a container via an MCP[1], propagate any changes the agent makes to git branches on the host, for easy review & merge. [0]: https://github.com/dagger/container-use https://github.com/dagger/container-use [1]: I would like it better if the entire agent process were sandboxed but I suppose that would make it difficult to use with IDE/GUI-based agents.
warthog 1y ago
I built this using OpenAI GPT-5 agentic loops as well where the agent interacts with a spreadsheet: banker dot so Some principles I learned: 1. You don't really wanna use any kind of framework that wraps around openai or claude sdks. You end up fighting them 2. Small number of tools with more functions are better than large number of tools with small number of functions inside 3. Function definitions you put inside the tools are actually very important and need to be very clear 4. Integrating another agent as a tool to the main agent can relieve your context window by 5x 5. RAG via only vector search is mostly unnecessary. Claude Code itself leverages iterative search which works way better with code repos as well as most documents
vinhnx 1y ago
I'm currently building my own coding agent, mainly for self-research [0], so this article is very helpful to me. Thank you Simon! In this particular thread, other has commented about lightweight sandboxing solution. I think I will integrate this next in my agent for safe code execution. [1] [0] https://github.com/vinhnx/vtcode https://github.com/vinhnx/vtcode [1] https://news.ycombinator.com/item?id=45429787 https://news.ycombinator.com/item?id=45429787

284 points by simonw 1y ago

I updated this post to link to the Claude Code docs that suggest running YOLO mode using their Docker dev container: https://www.anthropic.com/engineering/claude-code-best-practices#d-safe-yolo-mode https://www.anthropic.com/engineering/claude-code-best-pract... - which locks down network access to just a small set of domains: https://github.com/anthropics/claude-code/blob/5062ed93fc67f9322f807ecbf391ae4376cf8e83/.devcontainer/init-firewall.sh#L67-L76 https://github.com/anthropics/claude-code/blob/5062ed93fc67f...

saltyoldman 1y ago

I wouldn't be surprised if agents start getting managed by a distributed agentic system - think about it. Right now you get codex/claude/etc... and it's system prompt and various other internally managed prompts are locked down to the version you downloaded. What if a distributed system ran experimental prompts and monitored the success rate (what code makes it into a commit) and provides feedback to the agent manager. That could help automatically fine tune it's own prompts.

datadrivenangel 1y ago
This is what Anthropic does for their "High Compute" SWE benchmarking: " For our "high compute" numbers we adopt additional complexity and parallel test-time compute as follows: - We sample multiple parallel attempts. - We discard patches that break the visible regression tests in the repository, similar to the rejection sampling approach adopted by Agentless (Xia et al. 2024); note no hidden test information is used. - We then use an internal scoring model to select the best candidate from the remaining attempts. - This results in a score of 82.0% for Sonnet 4.5. " But running in parallel means you use more compute, meaning that the cost is higher. Good results are worth paying for, but if a super recursive approach costs more and takes longer than a human...

mccoyb 1y ago

I think this is a strictly worse name than "agentic harness", which is already a term used by open-source agentic IDEs (https://github.com/search?q=repo%3Aopenai%2Fcodex%20harness&type=code https://github.com/search?q=repo%3Aopenai%2Fcodex%20harness&... or https://github.com/openai/codex/discussions/1174 https://github.com/openai/codex/discussions/1174) Any reason why you want to rename it? Edit: to say more about my opinions, "agentic loop" could mean a few things -- it could mean the thing you say, or it could mean calling multiple individual agents in a loop ... whereas "agentic harness" evokes a sort of interface between the LLM and the digital outside world which mediates how the LLM embodies itself in that world. That latter thing is exactly what you're describing, as far as I can tell.

simonw 1y ago
I like "agentic harness" too, but that's not the name of a skill. "Designing agentic loops" describes a skill people need to develop. "Designing agentic harnesses" sounds more to me like you're designing a tool like Claude Code from scratch. Plus "designing agentic loops" includes a reference to my preferred definition of the term "agent" itself - a thing that runs tools in a loop to achieve a goal.
- prats226 1y ago
  Context engineering is another name people have given to same skill?
  - simonw 1y ago
    I think that's actually quite different. Context engineering is about making sure you've stuffed the context with all of the necessary information - relevant library documentation and examples and suchlike. Design the agentic loop is about picking the right tools to be provided to the model. The tool descriptions may go in the context but you also need to provide the right implementations of them.
    - tptacek 1y ago

deleted 1y ago

[deleted]

mike_hearn 1y ago

I recently built my own coding agent, due to dissatisfaction with the ones that are out there (though the Claude Code UI is very nice). It works as suggested in the article. It starts a custom Docker container and asks the model, GPT-5 in this case, to send shell scripts down the wire which are then run in the container. The container is augmented with some extra CLI tools to make the agent's life easier. My agent has a few other tricks up its sleeve and it's very new, so I'm still experimenting with lots of different ideas, but there are a few things I noticed. One is that GPT-5 is extremely willing to speculate. This is partly because of how I prompt it, but it's willing to write scripts that try five or six things at once in a single script, including things like reading files that might not exist. This level of speculative execution speeds things up dramatically especially as GPT-5 is otherwise a very slow model that likes to think about things a lot. Another is that you can give it very complex "missions" and it will drive things to completion using tactics that I've not seen from other agents. For example, if it needs to check something that's buried in a library dependency, it'll just clone the upstream repository into its home directory and explore that to find what it needs before going back to working on the user's project. None of this triggers any user interaction due to running in the container. In fact, no user interaction is possible. You set it going and do something else until it finishes. The model is very much to queue up "missions" that can then run in parallel and you merge them together at the end. The agent also has a mode where it takes the mission, writes a spec, reviews the spec, updates the spec given the review, codes, reviews the code, etc. Even though it's early days I've set this agent missions that it spent 20 minutes of continuous uninterrupted inferencing time on, and succeeded excellently. I think this UI paradigm is the way to go. You can't scale up AI assisted coding if you're constantly needing to interact with the agent. Getting the most out of models requires maximally exploiting parallelism, so sandboxing is a must.

simonw 1y ago
That's really interesting. I've noticed something similar - I've tried frontend tasks against GPT-5-Codex and seen it guess the URL of the underlying library (on jsdelivr or GitHub) and attempt to fetch the original source code, often trying several different URLs, in order to dig through the source and figure out how to use an undocumented API feature.

akashakya 1y ago

For lightweight sandboxing on Linux you can use bubblewrap or firejail instead of Docker. They are faster and _simpler_. Here is a bwrap script I wrote to run Claude in a minimal sandbox an hour back: exec bwrap \ --ro-bind /usr /usr \ --ro-bind /etc /etc \ --ro-bind /run /run \ --ro-bind "$NODE_PATH" /node \ --proc /proc \ --dev /dev \ --symlink usr/lib64 /lib64 \ --tmpfs /tmp \ --unshare-all \ --share-net \ --die-with-parent \ --new-session \ --bind "$HOME/claude" /claude \ --bind "$HOME/.claude.json" /claude/.claude.json \ --bind "$HOME/.claude" /claude/.claude \ --setenv HOME /claude \ --setenv PATH "/node:/claude/bin:/usr/bin" \ --bind "$(pwd)" /work \ --chdir /work \ /claude/bin/claude "$@"

simonw 1y ago
Nice, thanks for sharing. The lack of an equivalent on macOS (sandbox-exec is similar but mostly undocumented and described as "deprecated" by Apple) is really frustrating.
- DowsingSpoon 1y ago
  I had been planning to explore Lima tonight as a mechanism to shackle CC on macOS. The trouble with sandbox-exec is that it’s control over network access is not fine grain enough, and I found its file system controls insufficient. Also, I recently had some bad experiences which lead me to believe the tool MUST be run with strict CPU and memory resource limits, which is tricky on macOS.
  - simonw 1y ago
    Neat, I've not tried https://github.com/lima-vm/lima https://github.com/lima-vm/lima
  - big_toast 1y ago
    Wait, does lima do isolation in a macos context too? It looks like linux vms, which apple's container-cli (among others) covers at a basic level. I'd like apple to start providing macOS images that weren't the whole OS.. unless sandbox-exec/libsandbox have affordance for something close enough? You can basically ask claude/chatgpt to write its jail (dockerfile) and then run that via `container` without installing anything on macos outside the container it builds (IIRC). Even the container-cli will use a container to build your container..

tra3 1y ago

On one hand, folks are complaining that even basic tasks are impossible with LLMs. On the other hand we have cursed language [0] which was fully driven by AI and seems to be functional. Btw, how much did that cost? I feel like I've been hugely successful tools like Claude Code, aider, open code. Especially when I can define custom tools. "You" have to be a part of the loop, in some capacity. To provide guidance and/or direction. I'm puzzled by the fact that people are surprised by this. When I'm working with other entities (people) who are not artificially intelligent, the majority of the time is spent clarifying requirements and aligning on goals. Why would it be different with LLMs? 0: https://ghuntley.com/cursed/ https://ghuntley.com/cursed/

simonw 1y ago
The cursed cost estimate was $14,000 https://twitter.com/GeoffreyHuntley/status/1965295152962097550 https://twitter.com/GeoffreyHuntley/status/19652951529620975...
rv64imafdc 1y ago
> Btw, how much did that cost? $30k+ Not sure how functional though!

tptacek 1y ago

This is great. I feel like most of the oxygen is going to go to the sandboxing question (fair enough). But I'm kind of obsessed with what agent loops for engineering tasks that aren't coding look like, and also the tweaks you need for agent loops that handle large amounts of anything (source code lines, raw metrics or oTel span data, whatever). There was an interval where the notion of "context engineering" came into fashion, and we quickly dunked all over it (I don't blame anybody for that; "prompt engineering" seemed pretty cringe-y to me), but there's definitely something to the engineering problems of managing a fixed-size context window while iterating indefinitely through a complex problem, and there's all sorts of tricks for handling it.

kridsdale1 1y ago
I use agentic loops for document and spreadsheet (aka bullshit to interface with non engineers in the company) since my model of choice is adept at markdown and csv, and the office software can import and export those.
m4r71n 1y ago
I can imagine an agentic loop that updates dependencies à la Dependabot/Renovate-style by going through the changelog of a new version, reviewing new code changes, and evaluating whether it's worth it to upgrade (or even dangerous to do so, either from stability or security point of view). Too often these tools are used to blindly respin builds with the latest and greatest versions, which is what gets most people in trouble when their NPM deps become malicious.
- tptacek 1y ago
  You should write it! You might be surprised how easy it is to get something working.
  - m4r71n 1y ago
    Ha, well look at that, not even a day later: https://fossa.com/blog/fossabot-dependency-upgrade-ai-agent/ https://fossa.com/blog/fossabot-dependency-upgrade-ai-agent/

CuriouslyC 1y ago

One important issue with agentic loops is that agents are lazy, so you need some sort of retrigger mechanism. Claude code supports hooks, you can wire your agent stop hook to a local LLM, feed the context in and ask the model to prompt Claude to continue if needed. It works pretty well, Claude can override retriggers if it's REALLY sure it's done. Regarding sandboxing, VMs are the way. Prompt injected agents WILL be able to escape containers 100%.

tptacek 1y ago
Seems "big if true" given the number of cloud providers that use containers for customer isolation.
- CuriouslyC 1y ago
  This is not speculative, it's happened plenty already. People put mitigations in place, patch libraries and move on. The difference is that agents will find new zero days you've never heard of for stuff on your system people haven't scrutinized adequately. There will be zero advanced notice, and unlike human attackers who need to lie low until they can plan an exit, it'll be able to exploit you heavily right away. Do not take the security impact of agents lightly!
  - tptacek 1y ago
    I feel like my bona fides on this topic are pretty solid (without getting into my background on container vs. VM vs. runtime isolation) and: "the agents will find new zero days" also seems "big if true". I point `claude` at a shell inside a container and tell it "go find a zero day that breaks me out of this container", and you think I'm going to succeed at that? I had assumed you were saying something more like "any attacker that prompt-injects you probably has a container escape in their back pocket they'll just stage through the prompt injection vector", but you apparently meant something way further out.
    - CuriouslyC 1y ago

jujugoboom 1y ago

Wrong takeaway, but claude code was released February of this year??? I swear people have been glazing it for way longer... my memory isn't that bad right?

simonw 1y ago
February 24th, Anthropic tucked it into the same announcement as Sonnet 3.7: https://www.anthropic.com/news/claude-3-7-sonnet https://www.anthropic.com/news/claude-3-7-sonnet

refset 1y ago

For anyone else curious about what a practical loop implementation might look like, Steve Yegge YOLO-bootstrapped his 'Efrit' project using a few lines of Elisp: https://github.com/steveyegge/efrit/blob/4feb67574a330cc789fece2d84aeb5e2964f3537/archived/efrit-agent.el#L495-L536 https://github.com/steveyegge/efrit/blob/4feb67574a330cc789f... And for more context on Efrit this is a fun watch: "When Steve Gives Claude Full Access To 50 Years of Emacs Capabilities" https://www.youtube.com/watch?v=ZJUyVVFOXOc https://www.youtube.com/watch?v=ZJUyVVFOXOc

ademup 1y ago

I'm surprised to see so many people using containers when setting up a KVM is so easy, gives the most robust environment possible, and to my knowledge much has better isolation. A vanilla build of Linux plus your IDE of choice and you're off to the races.

scubbo 1y ago
Not everyone has spare hardware lying around!
- ademup 1y ago
  For sure! But just for reference, I'm on a mid-tier 2022 Dell Inspiron laptop: Ryzen 7 5825U with 64GB ram and 500GB SSD. On it, I run Ubuntu 24.04 as my host, and my guest is Lubuntu with 16GB ram and 80GB ssd for my KVM. I almost always have 2 instances of PHPstorm open in both Host and and Guest with multiple terminal tabs running various agentic tasks.
  - tecoholic 1y ago
    64 GB. Say no more :) I lived with 16GB until last year and upgraded to 32 only this year, which I thought was a huge improvement. I suspect a lot of people are around this ballpark, especially if they have bought Macs. Mine is Linux, still. So containers are the “simpler” versions.
  - scubbo 1y ago
    ...wait, you and I are using "KVM" in different ways, then. To me, it means a switch that lets you use the same Keyboard, Monitor ("Video"), and Mouse for two different machines. Sounds like you're talking instead about a technique for running a VM on a single machine - which, from Googlin', I suspect is "Kernel-based Virtual Machine", a new-to-me term. Thanks for teaching me something!
mike_hearn 1y ago
You often don't need strong isolation. The sandboxing is more to avoid model accidents than a Skynet scenario.

BoppreH 1y ago

> If anything goes wrong it’s a Microsoft Azure machine somewhere that’s burning CPU and the worst that can happen is code you checked out into the environment might be exfiltrated by an attacker, or bad code might be pushed to the attached GitHub repository. Isn't that risking getting banned from Azure? The compromised agent might not accomplish anything useful, but its attempts might get (correctly!) flagged by the cloud provider.

simonw 1y ago
My guess is that most cloud providers have procedures in place to help avoid banning legitimate customers because one of their instances got infected with malware (effectively what a rogue agent would be).

ggggggdgdhdjdj 1y ago

I find the word agentic to be a hot spike into my brain ... I am not sure why but I think it has to do with the fact it is marketing bs. Is it just me ?

handfuloflight 1y ago
It's marketing BS in the hands of grifters.
- simonw 1y ago
  Can you expand on why you think that?
  - handfuloflight 1y ago
    I'm saying there are grifters who co-opt the terms used by people actually pioneering / exploring the space, to sell BS; and that taints OP's view. This doesn't need stating but if it does I am not saying you are a grifter, obviously, because you substantiate your positions. Just another classic signal v. noise.
    - simonw 1y ago
      That makes sense, thanks.
- nobodywillobsrv 1y ago
  "Agentic" feels like the new "Generative". Yes, it means something "vague" and vibes oriented. It's when you can run things in a terminal or async or something maybe.
  - simonw 1y ago
    I've been trying to make it less vague. I'm using it to mean something that's directly related to running tools in a loop to achieve a goal.

moduspol 1y ago

A little off-topic but I’ve been waiting for a post by someone who’s built something like Claude Code or Codex that can explain how the state machine works for execution. Clearly it’s not just a good system prompt and hoping for the best. Obviously they’ve both got fairly good algorithms for working toward whatever the user asked for incrementally, handling expectable errors, changing course if necessary, etc. I mean ultimately it’s gotta be a really big flow chart that results in the LLM getting prompted various ways in all kinds of directions before ultimately converging on the final response, right? Or am I way off-base? Has anyone seen such a post explaining this architecture / state machine / algorithm?

zenburnmyface 1y ago
This is in that direction: https://jannesklaas.github.io/ai/2025/07/20/claude-code-agent-design.html https://jannesklaas.github.io/ai/2025/07/20/claude-code-agen...
- simonw 1y ago
  (This is a much better answer than my Codex Cloud generated version in the sibling comment.)
- moduspol 1y ago
  Thanks. Looks like exactly what I was looking for.
simonw 1y ago
Since Codex CLI is open source I got Codex Cloud to attempt to answer your question - here's the shared session: https://chatgpt.com/s/cd_68dc99fe3e948191a8923ddc4a1f4310 https://chatgpt.com/s/cd_68dc99fe3e948191a8923ddc4a1f4310 And here's the markdown file it produced with an attempt at an answer: https://github.com/simonw/codex-scratchpad/blob/codex/analyze-codex-source-code-architecture/answer.md https://github.com/simonw/codex-scratchpad/blob/codex/analyz...
mike_hearn 1y ago
You'd be surprised. You don't need a state machine to get started, only if you want to get really advanced. The models are smart. It's enough to just ask them to send you instructions, execute them and send back the results.

codazoda 1y ago

I made a simple docker container to jail an LLM for a recent project. https://github.com/codazoda/llm-jail https://github.com/codazoda/llm-jail

cadamsdotcom 1y ago

My preferred ergonomics for agents are like using a washing machine: choose what clothes you want to have cleaned (whites or colors..) and press “go”. Claude Code plan mode is the washing machine of agentic loops. It lets you see a detailed plan and if you approve you can let it rip with trust that it’ll do what it says, and go do something else while it’s at work. Adding tests and other ways to validate its work help it go further & run longer off a single plan.

chevman 1y ago

I feel like the AI industry is basically recreating "workflow engines" again and again and again, in different forms, with different input mechanisms.

jarboot 1y ago
Temporal.io has been good for orchestrating agentic workflows, or workflows in general

amluto 1y ago

The state of YOLO is embarrassing: > Update: It turns out Anthropic have their own documentation on Safe YOLO mode for Claude Code which says: > Letting Claude run arbitrary commands is risky and can result in data loss, system corruption, or even data exfiltration (e.g., via prompt injection attacks). To minimize these risks, use --dangerously-skip-permissions in a container without internet access. You can follow this reference implementation using Docker Dev Containers. [https://github.com/anthropics/claude-code/tree/main/.devcontainer https://github.com/anthropics/claude-code/tree/main/.devcont...] And… that link goes to a devcontainer that firewalls itself from inside using, effectively, sudo iptables. If Claude Code can’t break that all by itself in a single try, I’d be a bit surprised.

SafeDusk 1y ago

Resonating with the statement "Designing agentic loops is a very new skill—Claude Code was first released in just February 2025!" when I was investigating Codex[0]. [0]: https://blog.toolkami.com/openai-codex-tools/#coding-agents https://blog.toolkami.com/openai-codex-tools/#coding-agents

lobochrome 1y ago

Is there a way to let the agents share their agents.md? Claude code looks at Claude.md and I have yet to find a way to unify the agent handbook.

simonw 1y ago
At this point effectively everyone else is on https://agents.md/ https://agents.md/ - Claude are the only holdout.
- data-ottawa 1y ago
  Is there any reason this is better than just reading your regular readme or docs? All of these things look helpful for humans too!
  - threecheese 1y ago
    You could say that a Readme is declarative, while an agent file is imperative.
  - simonw 1y ago
    The main difference between AGENTS.md and a README.md is that coding agents will automatically ingest AGENTS.md when they start. You may not want them to ingest README.md if it's long and contains irrelevant information as that might be a waste of valuable tokens.
mike_hearn 1y ago
I just use symlinks.

Escapade5160 1y ago

I typically run my code agents in distrobox.

athrowaway3z 1y ago

On unix you can just use a user account to limit access. Create a group for you and the `claude` user, and have it do `umask 002` on login. The shellagent use case has much more parallels with 1980s TTY+mainframes dealing with multiple users than it has with containers. I.e. Docker's USP was reproducibility; not access controls.

mattacular 1y ago

> This is so dangerous, but it’s also key to getting the most productive results where do i sign up?

dboreham 1y ago

In case anyone else is confused: I saw the word "loop" and wondered what the point is to run an LLM in a loop? Like, what's the goal here? You want to write N different React todo web apps, or something? After reading the article a bit, I realized this isn't about "loops" at all. What it is about is "bypassing the yes/no human check on each step the LLM wants to take in completing some multi-step task". You can already to this with, for example, claude code. However, one of the points of having a human approve LLM actions is to prevent it doing something stupid or destructive. So now you have a second problem: how to prevent the LLM from poking holes in spacetime? Answer: sandbox it. And that's what the article is about.

deevus 1y ago

Fantastic article. I'm always looking for better ways to work with agents. I am personally using Claude Code, and want to try some more YOLO type stuff. My problem is I often give it too much to do. I'm usually overly optimistic about how effective it can be. I have found Sonnet 4.5 to be quite the improvement over Sonnet 4 however.

deleted 1y ago

[deleted]

codethief 1y ago

For sandboxing and parallelizing agents people at my day job recently started using container-use[0] and seem to be quite happy with it. I have yet to try it in depth but the approach it takes seems quite sensible to me: Have the agent execute all commands within a container via an MCP[1], propagate any changes the agent makes to git branches on the host, for easy review & merge. [0]: https://github.com/dagger/container-use https://github.com/dagger/container-use [1]: I would like it better if the entire agent process were sandboxed but I suppose that would make it difficult to use with IDE/GUI-based agents.

warthog 1y ago

I built this using OpenAI GPT-5 agentic loops as well where the agent interacts with a spreadsheet: banker dot so Some principles I learned: 1. You don't really wanna use any kind of framework that wraps around openai or claude sdks. You end up fighting them 2. Small number of tools with more functions are better than large number of tools with small number of functions inside 3. Function definitions you put inside the tools are actually very important and need to be very clear 4. Integrating another agent as a tool to the main agent can relieve your context window by 5x 5. RAG via only vector search is mostly unnecessary. Claude Code itself leverages iterative search which works way better with code repos as well as most documents

vinhnx 1y ago

I'm currently building my own coding agent, mainly for self-research [0], so this article is very helpful to me. Thank you Simon! In this particular thread, other has commented about lightweight sandboxing solution. I think I will integrate this next in my agent for safe code execution. [1] [0] https://github.com/vinhnx/vtcode https://github.com/vinhnx/vtcode [1] https://news.ycombinator.com/item?id=45429787 https://news.ycombinator.com/item?id=45429787