4 ms·

I recently built my own coding agent, due to dissatisfaction with the ones that are out there (though the Claude Code UI is very nice). It works as suggested in

I recently built my own coding agent, due to dissatisfaction with the ones that are out there (though the Claude Code UI is very nice). It works as suggested in the article. It starts a custom Docker container and asks the model, GPT-5 in this case, to send shell scripts down the wire which are then run in the container. The container is augmented with some extra CLI tools to make the agent's life easier. My agent has a few other tricks up its sleeve and it's very new, so I'm still experimenting with lots of different ideas, but there are a few things I noticed. One is that GPT-5 is extremely willing to speculate. This is partly because of how I prompt it, but it's willing to write scripts that try five or six things at once in a single script, including things like reading files that might not exist. This level of speculative execution speeds things up dramatically especially as GPT-5 is otherwise a very slow model that likes to think about things a lot. Another is that you can give it very complex "missions" and it will drive things to completion using tactics that I've not seen from other agents. For example, if it needs to check something that's buried in a library dependency, it'll just clone the upstream repository into its home directory and explore that to find what it needs before going back to working on the user's project. None of this triggers any user interaction due to running in the container. In fact, no user interaction is possible. You set it going and do something else until it finishes. The model is very much to queue up "missions" that can then run in parallel and you merge them together at the end. The agent also has a mode where it takes the mission, writes a spec, reviews the spec, updates the spec given the review, codes, reviews the code, etc. Even though it's early days I've set this agent missions that it spent 20 minutes of continuous uninterrupted inferencing time on, and succeeded excellently. I think this UI paradigm is the way to go. You can't scale up AI assisted coding if you're constantly needing to interact with the agent. Getting the most out of models requires maximally exploiting parallelism, so sandboxing is a must.

simonw 1y ago
That's really interesting. I've noticed something similar - I've tried frontend tasks against GPT-5-Codex and seen it guess the URL of the underlying library (on jsdelivr or GitHub) and attempt to fetch the original source code, often trying several different URLs, in order to dig through the source and figure out how to use an undocumented API feature.
- mike_hearn 1y ago
  Yes. It made me realize how much intelligence is in these models that isn't being exploited due to minor details of the harness. I've been doing this as a side project and it took nearly no effort to get something that I felt worked better than every other agent I tried, even if the UI is rougher. We're really in the stone age with this stuff. The models are not the limiting factor.
tptacek 1y ago
This also feels like the structure that Sketch.dev uses --- it's asynchronous running in a YOLO mode in a container on a cloud instance, with very little interaction (the expectation is you give it tasks and walk away). I have friends that queue up lots of tasks in the morning and prune down to just a couple successes in the afternoon. I'd do this too but for the scale I work on merge conflicts are too problematic. I'm working on my own dumb agent for my own dumb problems (engineering, but not software development) and I'd love to hear more about the tricks you're spotting.
- mike_hearn 1y ago
  Yes it's a bit like that except not in the cloud. It runs locally and doesn't make PRs, it just leaves your worktree in an uncommitted state so you can do any final tests, fixes to the code etc before committing.
- danenania 1y ago
  What kind of non-software engineering, if you don't mind sharing? Sounds interesting!
jsnell 1y ago
> Getting the most out of models requires maximally exploiting parallelism, so sandboxing is a must. What are your thoughts on checkpointing as a refinement of sandboxing? For tight human/llm loops I find automatic checkpoints (of both model context and file system state) and easy rolling back to any checkpoint to the most important tool. It's just so much faster to roll back on major mistakes and try again with the proper context, than to try to get the LLM to fix a mistake, since now the broken code and invalid assumptions are contaminating the context. But that relies on the human in the loop deciding when to undo. Are you giving some layer of your system the power of resetting sub-agents to previous checkpoints, do you do a full mind-wipe of the sub-agents if they get stuck and try again, or is the context rot just not a problem in practice?
- mike_hearn 1y ago
  I want to minimize human in the loop. At the moment my agent allows user interaction in one place, after a spec is written+reviewed+updated to reflect the review, it stops. You can then edit the spec before asking for an implementation. It helps to catch cases where the instructions were ambiguous. At the moment my agent is pretty basic. It doesn't detect endless loops. The model is allowed to bail at any time when it feels it's done or is stuck, so it doesn't seem to need to. It doesn't checkpoint currently. If it does the wrong thing you just roll it all back and improve the AGENTS.md or the mission text. That way you're less likely to encounter problems next time. The downside is that it's an expensive way to do things but for various reasons that's not a concern for this agent. One of the things I'm experimenting with is how very large token budgets affect agent design.
treyd 1y ago
Can you share a link to your repo? I am curious to understand the architecture here, and what the prompts you're using are like that enable GPT-5 to have that kind of emergent behavior.
- mike_hearn 1y ago
  It's private unfortunately, I doubt it'll be available publicly anytime soon. I don't own the copyright on the code, but even if I did, I don't feel like entering the agent race right now. That's becoming all about who converts VC capital into tokens most efficiently and I'm not sure how to build a sustainable business out of it given what looks like systematic market dumping. I wouldn't worry. There's so much attention focused on this space right now that any ideas I have are being had simultaneously by hundreds of other people. I just discovered a tool called Sculptor that's got a pretty UI of the type I wanted for the container management; they are probably the ones to watch for this kind of agentic approach.
nobodywillobsrv 1y ago
What kind of tasks were successful? What kinds of tasks failed? Have you tried deeply "get me the data" tasks say for example "assemble me the time series of cash flows associated with the Japanese Soverieng Debt". Something I have been struggling with is that for "go and look and find this and join it and then see" kind of exercises really seem to go way way off the rails and Claude in particular likes to "mock" anything that gets a little bit hard even if I scream "NEVER MOCK DATA" in my agents file. I would be curious if there are tricks in these long running loops like constantly injecting new "DO NOT MOCK" shouts at it every n cycles or something.
- mike_hearn 1y ago
  I've only tried it with various programming tasks. An example task: port the docs for this framework to a different (programming) language, touching up the text along the way for the new ecosystem and rewriting the code samples. I haven't tried data analysis tasks.

simonw 1y ago

That's really interesting. I've noticed something similar - I've tried frontend tasks against GPT-5-Codex and seen it guess the URL of the underlying library (on jsdelivr or GitHub) and attempt to fetch the original source code, often trying several different URLs, in order to dig through the source and figure out how to use an undocumented API feature.

mike_hearn 1y ago
Yes. It made me realize how much intelligence is in these models that isn't being exploited due to minor details of the harness. I've been doing this as a side project and it took nearly no effort to get something that I felt worked better than every other agent I tried, even if the UI is rougher. We're really in the stone age with this stuff. The models are not the limiting factor.

tptacek 1y ago

This also feels like the structure that Sketch.dev uses --- it's asynchronous running in a YOLO mode in a container on a cloud instance, with very little interaction (the expectation is you give it tasks and walk away). I have friends that queue up lots of tasks in the morning and prune down to just a couple successes in the afternoon. I'd do this too but for the scale I work on merge conflicts are too problematic. I'm working on my own dumb agent for my own dumb problems (engineering, but not software development) and I'd love to hear more about the tricks you're spotting.

mike_hearn 1y ago
Yes it's a bit like that except not in the cloud. It runs locally and doesn't make PRs, it just leaves your worktree in an uncommitted state so you can do any final tests, fixes to the code etc before committing.
danenania 1y ago
What kind of non-software engineering, if you don't mind sharing? Sounds interesting!

jsnell 1y ago

> Getting the most out of models requires maximally exploiting parallelism, so sandboxing is a must. What are your thoughts on checkpointing as a refinement of sandboxing? For tight human/llm loops I find automatic checkpoints (of both model context and file system state) and easy rolling back to any checkpoint to the most important tool. It's just so much faster to roll back on major mistakes and try again with the proper context, than to try to get the LLM to fix a mistake, since now the broken code and invalid assumptions are contaminating the context. But that relies on the human in the loop deciding when to undo. Are you giving some layer of your system the power of resetting sub-agents to previous checkpoints, do you do a full mind-wipe of the sub-agents if they get stuck and try again, or is the context rot just not a problem in practice?

mike_hearn 1y ago
I want to minimize human in the loop. At the moment my agent allows user interaction in one place, after a spec is written+reviewed+updated to reflect the review, it stops. You can then edit the spec before asking for an implementation. It helps to catch cases where the instructions were ambiguous. At the moment my agent is pretty basic. It doesn't detect endless loops. The model is allowed to bail at any time when it feels it's done or is stuck, so it doesn't seem to need to. It doesn't checkpoint currently. If it does the wrong thing you just roll it all back and improve the AGENTS.md or the mission text. That way you're less likely to encounter problems next time. The downside is that it's an expensive way to do things but for various reasons that's not a concern for this agent. One of the things I'm experimenting with is how very large token budgets affect agent design.

treyd 1y ago

Can you share a link to your repo? I am curious to understand the architecture here, and what the prompts you're using are like that enable GPT-5 to have that kind of emergent behavior.

mike_hearn 1y ago
It's private unfortunately, I doubt it'll be available publicly anytime soon. I don't own the copyright on the code, but even if I did, I don't feel like entering the agent race right now. That's becoming all about who converts VC capital into tokens most efficiently and I'm not sure how to build a sustainable business out of it given what looks like systematic market dumping. I wouldn't worry. There's so much attention focused on this space right now that any ideas I have are being had simultaneously by hundreds of other people. I just discovered a tool called Sculptor that's got a pretty UI of the type I wanted for the container management; they are probably the ones to watch for this kind of agentic approach.

nobodywillobsrv 1y ago

What kind of tasks were successful? What kinds of tasks failed? Have you tried deeply "get me the data" tasks say for example "assemble me the time series of cash flows associated with the Japanese Soverieng Debt". Something I have been struggling with is that for "go and look and find this and join it and then see" kind of exercises really seem to go way way off the rails and Claude in particular likes to "mock" anything that gets a little bit hard even if I scream "NEVER MOCK DATA" in my agents file. I would be curious if there are tricks in these long running loops like constantly injecting new "DO NOT MOCK" shouts at it every n cycles or something.

mike_hearn 1y ago
I've only tried it with various programming tasks. An example task: port the docs for this framework to a different (programming) language, touching up the text along the way for the new ecosystem and rewriting the code samples. I haven't tried data analysis tasks.