13 ms·

The unreasonable effectiveness of an LLM agent loop with tool use

447 points by crawshaw 1y ago

andes314 1y ago
This is what the no-code API-to-MCP creator uses at usetexture.com! I was surprised to find out this is not what the Claude client uses (as of May 2025)
_bin_ 1y ago
I've found sonnet-3.7 to be incredibly inconsistent. It can do very well but has a strong tendency to get off-track and run off and do weird things. 3.5 is better for this, ime. I hooked claude desktop up to an MCP server to fake claude-code less the extortionate pricing and it works decently. I've been trying to apply it for rust work; it's not great yet (still doesn't really seem to "understand" rust's concepts) but can do some stuff if you make it `cargo check` after each change and stop it if it doesn't. I expect something like o3-high is the best out there (aider leaderboards support this) either alone or in combination with 4.1, but tbh that's out of my price range. And frankly, I can't mentally get past paying a very high price for an LLM response that may or may not be useful; it leaves me incredibly resentful as a customer that your model can fail the task, requiring multiple "re-rolls", and you're passing that marginal cost to me.
- agilebyte 1y ago
  I am avoiding the cost of API access by using the chat/ui instead, in my case Google Gemini 2.5 Pro with the high token window. Repomix a whole repo. Paste it in with a standard prompt saying "return full source" (it tends to not follow this instruction after a few back and forths) and then apply the result back on top of the repo (vibe coded https://github.com/radekstepan/apply-llm-changes https://github.com/radekstepan/apply-llm-changes to help me with that). Else yeah, $5 spent on Cline with Claude 3.7 and instead of fixing my tests, I end up with if/else statements in the source code to make the tests pass.
  - harvey9 1y ago
    Guess it was trained by scraping thedailywtf.com
  - actsasbuffoon 1y ago
    I decided to experiment with Claude Code this month. The other day it decided the best way to fix the spec was to add a conditional to the test that causes it to return true before getting to the thing that was actually supposed to be tested. I’m finding it useful for really tedious stuff like doing complex, multi step terminal operations. For the coding… it’s not been great.
    - nico 1y ago
      I’ve had this in different ways many times. Like instead of resolving the underlying issue for an exception, it just suggests catching the exception and keep going It also depends a lot on the mix of model and type of code and libraries involved. Even in different days the models seem to be more or less capable (I’m assuming they get throttled internally - this is very noticeable sometimes in how they try to save on output tokens and summarize the code responses as much as possible, at least in the chat/non-api interfaces)
    - christophilus 1y ago
      Well, that’s proof that it used my GitHub projects in its training data.
  - nico 1y ago
    Cool tool. What format does it expect from the model? I’ve been looking for something that can take “bare diffs” (unified diffs without line numbers), from the clipboard and then apply them directly on a buffer (an open file in vscode) None of the paste diff extension for vscode work, as they expect a full unified diff/patch I also tried a google-developed patch tool, but also wasn’t very good at taking in the bare diffs, and def couldn’t do clipboard
    - agilebyte 1y ago
- layoric 1y ago
  I've been using Mistral Medium 3 last couple of days, and I'm honestly surprised at how good it is. Highly recommend giving it a try if you haven't, especially if you are trying to reduce costs. I've basically switched from Claude to Mistral and honestly prefer it even if costs were equal.
  - nico 1y ago
    How are you running the model? Mistral’s api or some local version through ollama, or something else?
    - kyleee 1y ago
      Is mistral on open router?
      - nico 1y ago
        Yup https://openrouter.ai/provider/mistral https://openrouter.ai/provider/mistral I guess it can't really be run locally https://www.reddit.com/r/LocalLLaMA/comments/1kgyfif/introducing_mistral_medium_3/ https://www.reddit.com/r/LocalLLaMA/comments/1kgyfif/introdu...
    - layoric 1y ago
      Through OpenRouter, medium 3 isn't open weights.
- johnsmith1840 1y ago
  I seem to be alone in this but the only methods truly good at coding are slow heavy test time compute models. o1-pro and o1-preview are the only models I've ever used that can reliably update and work with 1000 LOC without error. I don't let o3 write any code unless it's very small. Any "cheap" model will hallucinate or fail massively when pushed. One good tip I've done lately. Remove all comments in your code before passing or using LLMs, don't let LLM generated comments persist under any circumstance.
  - _bin_ 1y ago
    Interesting. I've never tested o1-pro because it's insanely expensive but preview seemed to do okay. I wouldn't be shocked if huge, expensive-to-run models performed better and if all the "optimized" versions were actually labs trying to ram cheaper bullshit down everyone's throat. Basically chinesium for LLMs; you can afford them but it's not worth it. I remember someone saying o1 was, what, 200B dense? I might be misremembering.
    - johnsmith1840 1y ago
      I'm positive they are pushing users to cheaper models due to cost. o1-pro is now in a sub menu for pro users and labled legacy. The big inference methods must be stupidly expensive. o1-preview was and possibly still is the most powerful model they ever released. I only switched to pro for coding after months of them improving it and my api bill getting a bit crazy (like 0.50$ per question). I don't think paramater count matters anymore. I think the only thing that matters is how much compute a vendor will give you per question.
  - doug_durham 1y ago
    I never have LLMs work on 1000 LOC. I don't think that's the value-add. Instead I use it a the function and class level to accelerate my work. The thought of having any agent human or computer run amok in my code makes me uncomfortable. At the end of the day I'm still accountable for the work, and I have to read and comprehend everything. If do it piecewise I it makes tracking the work easier.
jbellis 1y ago
Yes! Han Xiao at Jina wrote a great article that goes into a lot more detail on how to turn this into a production quality agentic search: https://jina.ai/news/a-practical-guide-to-implementing-deepsearch-deepresearch/ https://jina.ai/news/a-practical-guide-to-implementing-deeps... This is the same principle that we use at Brokk for Search and for Architect. (https://brokk.ai/ https://brokk.ai/) The biggest caveat: some models just suck at tool calling, even "smart" models like o3. I only really recommend Gemini Pro 2.5 for Architect (smart + good tool calls); Search doesn't require as high a degree of intelligence and lots of models work (Sonnet 3.7, gpt-4.1, Grok 3 are all fine).
- crawshaw 1y ago
  I'm curious about your experiences with Gemini Pro 2.5 tool calling. I have tried using it in agent loops (in fact, sketch has some rudimentary support I added), and compared with the Anthropic models I have had to actively reprompt Gemini regularly to make tool calls. Do you consider it equivalent to Sonnet 3.7? Has it required some prompt engineering?
  - jbellis 1y ago
    Confession time: litellm still doesn't support parallel tool calls with Gemini models [https://github.com/BerriAI/litellm/issues/9686 https://github.com/BerriAI/litellm/issues/9686] so we wrote our own "parallel tool calls" on top of Structured Output. It did take a few iterations on the prompt design but all of it was "yeah I can see why that was ambiguous" kinds of things, no real complaints. GP2.5 does have a different flavor than S3.7 but it's hard to say that one is better or worse than the other [edit: at tool calling -- GP2.5 is definitely smarter in general]. GP2.5 is I would say a bit more aggressive at doing "speculative" tool execution in parallel with the architect, e.g. spawning multiple search agent calls at the same time, which for Brokk is generally a good thing but I could see use cases where you'd want to dial that back.
- sagarpatil 1y ago
  “Claude Code, better than Sourcegraph, better than Augment Code.” That’s a pretty bold claim, how come you are not at the top of this list then? https://www.swebench.com/ https://www.swebench.com/ “Use frontier models like o3, Gemini Pro 2.5, Sonnet 3.7” Is this unlimited usage? Or number of messages/tokens? Why do you need a separate desktop app? Why not CLI or VS Code extension.
kgeist 1y ago
Today I tried "vibe-coding" for the first time using GPT-4o and 4.1. I did it manually - just feeding compilation errors, warnings, and suggestions in a loop via the canvas interface. The file was small, around 150 lines. It didn't go well. I started with 4o: - It used a deprecated package. - After I pointed that out, it didn't update all usages - so I had to fix them manually. - When I suggested a small logic change, it completely broke the syntax (we're talking "foo() } return )))" kind of broken) and never recovered. I gave it the raw compilation errors over and over again, but it didn't even register the syntax was off - just rewrote random parts of the code instead. - Then I thought, "maybe 4.1 will be better at coding" (as advertized). But 4.1 refused to use the canvas at all. It just explained what I could change - as in, you go make the edits. - After some pushing, I got it to use the canvas and return the full code. Except it didn't - it gave me a truncated version of the code with comments like "// omitted for brevity". That's when I gave up. Do agents somehow fix this? Because as it stands, the experience feels completely broken. I can't imagine giving this access to bash, sounds way too dangerous.
- skeeter2020 1y ago
  I had an even better experience. I asked to produce a small web app with a new-to-me framework: success! I asked to make some CSS changes to the UI; the app no longer builds.
- vFunct 1y ago
  Use Claude Sonnet with an IDE.
- 85392_school 1y ago
  Agents definitely fix this. When you can run commands and edit files, the agent can test its code by itself and fix any issues.
- visarga 1y ago
  You should try Cursor or Windsurf, with Claude or Gemini model. Create a documentation file first. Generate tests for everything. The more the better. Then let it cycle 100 times until tests pass. Normal programming is like walking, deliberate and sure. Vibe coding is like surfing, you can't control everything, just hit yes on auto. Trust the process, let it make mistakes and recover on its own.
libraryofbabel 1y ago
Strongly recommend this blog post too which is a much more detailed and persuasive version of the same point. The author actually goes and builds a coding agent from zero: https://ampcode.com/how-to-build-an-agent https://ampcode.com/how-to-build-an-agent It is indeed astonishing how well a loop with an LLM that can call tools works for all kinds of tasks now. Yes, sometimes they go off the rails, there is the problem of getting that last 10% of reliability, etc. etc., but if you're not at least a little bit amazed then I urge you go to and hack together something like this yourself, which will take you about 30 minutes. It's possible to have a sense of wonder about these things without giving up your healthy skepticism of whether AI is actually going to be effective for this or that use case. This "unreasonable effectiveness" of putting the LLM in a loop also accounts for the enormous proliferation of coding agents out there now: Claude Code, Windsurf, Cursor, Cline, Copilot, Aider, Codex... and a ton of also-rans; as one HN poster put it the other day, it seems like everyone and their mother is writing one. The reason is that there is no secret sauce and 95% of the magic is in the LLM itself and how it's been fine-tuned to do tool calls. One of the lead developers of Claude Code candidly admits this in a recent interview.[0] Of course, a ton of work goes into making these tools work well, but ultimately they all have the same simple core. [0] https://www.youtube.com/watch?v=zDmW5hJPsvQ https://www.youtube.com/watch?v=zDmW5hJPsvQ
- sesm 1y ago
  Should we change the link above to use `?utm_source=hn&utm_medium=browser` before opening it?
  - libraryofbabel 1y ago
    fixed :)
- kcorbitt 1y ago
  For "that last 10% of reliability" RL is actually working pretty well right now too! https://openpipe.ai/blog/art-e-mail-agent https://openpipe.ai/blog/art-e-mail-agent
- meander_water 1y ago
bhouston 1y ago
I found this out too - it is quite easy and effective: https://benhouston3d.com/blog/building-an-agentic-code-from-scratch https://benhouston3d.com/blog/building-an-agentic-code-from-...
simonw 1y ago
I'm very excited about tool use for LLMs at the moment. The trick isn't new - I first encountered it with the ReAcT paper two years ago - https://til.simonwillison.net/llms/python-react-pattern https://til.simonwillison.net/llms/python-react-pattern - and it's since been used for ChatGPT plugins, and recently for MCP, and all of the models have been trained with tool use / function calls in mind. What's interesting today is how GOOD the models have got at it. o3/o4-mini's amazing search performance is all down to tool calling. Even Qwen3 4B (2.6GB from Ollama, runs happily on my Mac) can do tool calling reasonably well now. I gave a workshop at PyCon US yesterday about building software on top of LLMs - https://simonwillison.net/2025/May/15/building-on-llms/ https://simonwillison.net/2025/May/15/building-on-llms/ - and used that as an excuse to finally add tool usage to an alpha version of my LLM command-line tool. Here's the section of the workshop that covered that: https://building-with-llms-pycon-2025.readthedocs.io/en/latest/tools.html https://building-with-llms-pycon-2025.readthedocs.io/en/late... My LLM package can now reliably count the Rs in strawberry as a shell one-liner: llm --functions ' def count_char_in_string(char: str, string: str) -> int: """Count the number of times a character appears in a string.""" return string.lower().count(char.lower()) ' 'Count the number of Rs in the word strawberry' --td
- andrewmcwatters 1y ago
  I love the odd combination of silliness and power in this.
- DarmokJalad1701 1y ago
  Was the workshop recorded?
  - simonw 1y ago
    No video or audio, just my handouts.
maxqper123 1y ago
[flagged]
mukesh610 1y ago
I built this very same thing today! The only difference is that i pushed the tool call outputs into the conversation history and resent it back to the LLM for it to summarize, or perform further tool calls, if necessary, automagically. I used ollama to build this and ollama supports tool calling natively, by passing a `tools=[...]` in the Python SDK. The tools can be regular Python functions with docstrings that describe the tool use. The SDK handles converting the docstrings into a format the LLM can recognize, so my tool's code documentation becomes the model's source of truth. I can also include usage examples right in the docstring to guide the LLM to work closely with all my available tools. No system prompt needed! Moreover, I wrote all my tools in a separate module, and just use `inspect.getmembers` to construct the `tools` list that i pass to Ollama. So when I need to write a new tool, I just write another function in the tools module and it Just Works™ Paired with qwen 32b running locally, i was fairly satisfied with the output.
- degamad 1y ago
  > The only difference is that i pushed the tool call outputs into the conversation history and resent it back to the LLM for it to summarize, or perform further tool calls, if necessary, automagically. It looks like this one does that too. msg = [ handle_tool_call(tc) for tc in tool_calls ]
  - mukesh610 1y ago
    Ah, failed to notice that. I was so excited because this was exactly what I coded up today, I jumped straight to the comments.
jawns 1y ago
Not only can this be an effective strategy for coding tasks, but it can also be used for data querying. Picture a text-to-SQL agent that can query database schemas, construct queries, run explain plans, inspect the error outputs, and then loop several times to refine. That's the basic architecture behind a tool I built, and I have been amazed at how well it works. There have been multiple times when I've thought, "Surely it couldn't handle THIS prompt," but it does! Here's an AWS post that goes into detail about this approach: https://aws.amazon.com/blogs/machine-learning/build-a-robust-text-to-sql-solution-generating-complex-queries-self-correcting-and-querying-diverse-data-sources/ https://aws.amazon.com/blogs/machine-learning/build-a-robust...
amelius 1y ago
Huh, isn't this already built-in, in most chat UIs?
- bkyan 1y ago
  This is for running locally in the terminal (or in WSL2 if you are on Windows). Here is an example session from a similar script that I <cough>ChatGPT</cough> wrote in Perl, illustrating the same concept: $ ~/agentloop.pl YOUR PROMPT: Is Python installed on this system? If so, which version? BASH STDIN: python --version 2>&1 || python3 --version 2>&1 BASH STDOUT: /bin/bash: python: command not found Python 3.6.7 LLM RESPONSE: Python is installed on this system. The version available is Python 3.6.7. However, it looks like the python executable (python) is not directly available, but python3 (or a direct version path) exists and resolves to Python 3.6.7. If you need to use Python, you should use python3 as the command.
- cheema33 1y ago
  Which chat UI lets me use my local tools? Like git, find, pnpm, curl...?
  - insin 1y ago
    Has anybody written an Electron app yet which injects tools into existing chat UIs, letting you you expose whatever you want from your machine to them via Node? I was planning to create something like that for the BigCo internal chat UI I work on.
    - energy123 1y ago
      Someone wrote a browser extension that does that for Gemini web chat
    - NitpickLawyer 1y ago
      Careful, this might break the ToS on some products. At least for those that offer all you can prompt in a browser vs. API.
BrandiATMuhkuh 1y ago
That's really cool. One week ago I implemented an SQL tool and it works really well. But sometimes it still just makes up table/column names. Luckily it can read the error and correct itself. But today I went to the next level. I gave the LLM two tools. One web search tool and one REST tool. I told it at what URL it can find API docs. Then I asked it to perform some tasks for me. It was really cool to watch an AI read docs, make api calls and try again (REPL) until it worked
rbren 1y ago
If you're interested in hacking on agent loops, come join us in the OpenHands community! Here's our (slightly more complicated) agent loop: https://github.com/All-Hands-AI/OpenHands/blob/f7cb2d0f64666e1f090a5152d7c002aa6f28caf9/openhands/controller/agent_controller.py#L771 https://github.com/All-Hands-AI/OpenHands/blob/f7cb2d0f64666...
- sagarpatil 1y ago
  I tried OH yesterday. Sorry to say this but your UI/UX experience is horrible. I don’t understand why do you need a separate UI instead of using local IDE (Cursor/Windsurf), vs code extension (augment) or CLI (Amazon Q developer). Please do not reinvent the wheel.
tqwhite 1y ago
I've been using Claude Code, ie, a terminal interface to Sonnet 3.7 since the day it came out in mid March. I have done substantial CLI apps, full stack web systems and a ton of utility crap. I am much more ambitious because of it, much as I was in the past when I was running a programming team. I'm sure it is much the same as this under the hood though Anthropic has added many insanely useful features. Nothing is perfect. Producing good code requires about the same effort as it did when I was running said team. It is possible to get complicated things working and find oneself in a mess where adding the next feature is really problematic. As I have learned to drive it, I have to do much less remediation and refactoring. That will never go away. I cannot imagine what happened to poor kgeist. I have had Claude make choices I wouldn't and do some stupid stuff, never enough that I would even think about giving up on it. Almost always, it does a decent job and, for a most stuff, the amount of work it takes off of my brain is IMMENSE. And, for good measure, it does a wonderful job of refactoring. Periodically, I have a session where I look at the code, decide how it could be better and instruct Claude. Huge amounts of complexity, done. "Change this data structure", done. It's amazingly cool. And, just for fun, I opened it in a non-code archive directory. It was a junk drawer that I've been filling for thirty years. "What's in this directory?" "Read the old resumes and write a new one." "What are my children's names?" Also amazing. And this is still early days. I am so happy.
- benoau 1y ago
  > And, for good measure, it does a wonderful job of refactoring. Periodically, I have a session where I look at the code, decide how it could be better and instruct Claude. Huge amounts of complexity, done. "Change this data structure", done. It's amazingly cool. Yeah this is literally just so enjoyable. Stuff that would be an up-hill battle to get included in a sprint takes 5 minutes. It makes it feel like a whole team is just sitting there, waiting to eagerly do my bidding with none of the headache waiting for work to be justified, scheduled, scoped, done, and don't even have to justify rejecting it if I don't like the results.
- felipeerias 1y ago
  Recently I had to define a remote data structure, specify the API to request it, implement parsing and storage, and show it to the user. Claude was able to handle all of these tasks simultaneously, so I could see how small changes at either end would impact the intermediate layers. I iterated on many ideas until I settled on the best overall solution for my use case. Being able to iterate like that through several layers of complexity was eye-opening. It made me more productive while giving me a better understanding of how the different components fit together.
kuahyeow 1y ago
What protection do people use when enabling an LLM to run `bash` on your machine ? Do you run it in a Docker container / LXC boundary ? `chroot` ?
- CGamesPlay 1y ago
  The blog post in question is on the site for Sketch, which appears to use Docker containers. That said, I use Claude Code, which just uses unsandboxed commands with manual approval. What's your concern? An accident or an attacker? For accidents, I use git and backups and develop in a devcontainer. For an attacker, bash just seems like an ineffective attack vector; I would be more worried about instructing the agent to write a reverse shell directly into the code.
  - zahlman 1y ago
    > For an attacker, bash just seems like an ineffective attack vector All it needs to do is curl and run the actual payload.
    - deleted 1y ago
      [deleted]
    - Wilder7977 1y ago
      Or even the standard "echo xxx | base64 -d" or a million other ways. How can someone say that bash is not interesting to an attacker is beyond me.
      - CGamesPlay 1y ago
        But bash isn't a key ingredient in any of these. The exact same payload could easily be insert in the project's source code, and has the benefit of being persistent. Using a bash shell to do it might be the most obvious way, sure, but shutting down bash access is such a poor defense that it isn't worth doing.
  - cess11 1y ago
outworlder 1y ago
> If you don't have some tool installed, it'll install it. Terrifying. LLMs are very 'accommodating' and all they need is someone asking them to do something. This is like SQL injection, but worse.
- rglover 1y ago
  I often wonder what the first agent-driven catastrophe will be. Considering the gold rush (emphasis on rush) going on, it's only a matter of time before a difficult to fix disaster occurs.
- nxobject 1y ago
  In an ideal world, this would require us as programmers to lean into our codebase reading and augmentation skills – currently underappreciated anyway as a skill to build. But when the incentives lean towards write-only code, I'm not optimistic.
cadamsdotcom 1y ago
> "Oh, this test doesn't pass... let's just skip it," it sometimes says, maddeningly. Here is a wild idea. Imagine running a companion, policy-enforcing LLM, independently and in parallel, which is given instructions to keep the main LLM behaving according to instructions. If the companion LLM could - in real time - ban the coding LLM from emitting "let's just skip it" by seeing the tokens "let's just" and then biasing the output such that the word "skip" becomes impossible to emit. Banning the word "skip" from following "let's just", forces the LLM down a new path away from the undesired behavior. It's like Structured Outputs or JSON mode, but driven by a companion LLM, and dynamically modified in real time as tokens are emitted. If the idea works, you could prompt the companion LLM to do more advanced stuff - eg. ban a coding LLM from making tests pass by deleting the test code, ban it from emitting pointless comments... all the policies that we put into system prompts today and pray the LLM will do, would go into the companion LLM's prompt instead. Wonder what the Outlines folks think of this!
- JoshuaDavid 1y ago
  Along these lines, if the main LLM goes down a bad path, you could _rewind_ the model to before it started going down the bad path -- the watcher LLM doesn't necessarily have to guess that "skip" is a bad token after the words "let's just", it could instead see "let's just skip the test" and go "nope, rolling back to the token "just " and rerolling with logit_bias={"skip":-10,"omit":-10,"hack":-10}". Of course doing that limits which model providers you can work with (notably, OpenAI has gotten quite hostile to power users doing stuff like that over the past year or so).
  - cadamsdotcom 1y ago
    That’s a really neat idea. Kind of seems an optimization: if the “token ban” is a tool call, you can see that being too slow to run for every token. Provided rewinding is feasible, your idea could make it performant enough to be practical.
    - TeMPOraL 1y ago
      It's not an optimization; the greedy approach won't work well, because it rejects critical context that comes after the tokens that would trigger the rejection. Consider: "Let's just skip writing this test, as the other change you requested seems to eliminate the need for it." Rolling back the model on "Let's just" would be stupid; rolling it on "Let's just skip writing this test" would be stupid too, as is the beliefs that writing tests is your holy obligation to your god, and you must do so unconditionally. The following context makes it clear that the decision is fine. Or, if you (or the governor agent) don't buy the reasoning, you're then in a perfect position to say, "nope, let's roll back to <Let's> and bias against ["skip, test"]". Checking the entire message once makes sense; checking it after every token doesn't.
magicalhippo 1y ago
Assuming the title is a play on the paper "The Unreasonable Effectiveness of Mathematics in the Natural Sciences"[1][2] by Eugene Wigner. [1]: https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness... [2]: https://www.hep.upenn.edu/~johnda/Papers/wignerUnreasonableEffectiveness.pdf https://www.hep.upenn.edu/~johnda/Papers/wignerUnreasonableE...
- gavmor 1y ago
  That may be its primogenitor, but it's long since become a meme: https://scholar.google.com/scholar?q=unreasonable+effectiveness https://scholar.google.com/scholar?q=unreasonable+effectiven...
  - -__---____-ZXyw 1y ago
    a. Primogenitor is a nice word! b. Wigner's original essay is a serious piece, and quite beautiful in its arguments. I had been under the impression that the phrasing had been used a few times since, but typically by other serious people who were aware of the lineage of that lovely essay. With this 6-paragraph vibey-blog-post, it truly has become a meme. So it goes, I suppose.
- dsubburam 1y ago
  I didn't know of that paper, and thought the title was a riff on Karpathy's Unreasonable Effectiveness of RNNs in 2015[1]. Even if my thinking is correct, as it very well might be given the connection RNNs->LLMs, Karpathy might have himself made his title a play on Wigner's (though he doesn't say so). [1] https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://karpathy.github.io/2015/05/21/rnn-effectiveness/
  - throwaway314155 1y ago
    Unreasonable effectiveness of [blah] has been a thing for decades if not centuries. It's not new.
leelou2 1y ago
[dead]
benoau 1y ago
This morning I used cursor to extract a few complex parts of my game prototype's "main loop", and then generate a suite of tests for those parts. In total I have 341 tests written by Cursor covering all the core math and other components. It has been a bit like herding cats sometimes, it will run away with a bad idea real fast, but the more constraints I give it telling it what to use, where to put it, giving it a file for a template, telling it what not to do, the better the results I get. In total it's given me 3500 lines of test code that I didn't need to write, don't need to fix, and can delete and regenerate if underlying assumptions change. It's also helped tune difficulty curves, generate mission variations and more.
- otterley 1y ago
  Writing tests, in my experience, is by far the best use case for LLMs. It eliminates hours or days of drudgery and toil, and can provide coverage of so many more edge cases that I can think of myself. Plus it makes your code more robust! It’s just wonderful all around.
  - benoau 1y ago
    Yeah I generated another 60 tests / 500 lines of test code since I posted my comment. I must have written 100,000s of lines of test codes the hard way, it's not just faster than me it's better than me when it comes to the coverage, and it requires less of my help than a number of developers I've had under me.
hbbio 1y ago
Yes, agent loops are simple, except, as the article says, a bit of "pump and circumstance"! If anyone is interested, I tried to put together a minimal library (no dependency) for TypeScript: https://github.com/hbbio/nanoagent https://github.com/hbbio/nanoagent
bdbenton5255 1y ago
Woke up this morning to start on a new project. Started with a math visualizer for machine learning, saw an HN post for this soon after and scrapped it. It was better done by someone else. Started on an LLM app that looped outputs, saw this post soon after and scrapped it. It was better done by someone else. It is like every single original notion I have is immediately done by someone else at the exact same time. I think I will just move on to rudimentary systems programming stuff and avoid creative and original thinking, just need basic and low profile employment.
- soulofmischief 1y ago
  What is your motive for creating things? Does it really matter if there is competition?
- throwaway314155 1y ago
  > Started on an LLM app that looped outputs, saw this post soon after and scrapped it. It was better done by someone else. If it helps, "TFA" was not the originator here and is merely simplifying concepts from fairly established implementations in the wild. As simonw mentions elsewhere, it goes back to at least the ReAct paper and maybe even more if you consider things like retrieval-augmented generation.
- morkalork 1y ago
  Everything has been done before, sometimes before you were even born. The trick is to just not google it first and do it anyways.
  - Lionga 1y ago
    No. DO google it, learn from it and make it better
themichaellai 1y ago
I've also been defining agents as "LLM call in a while loop with tools" to my co-workers as well — I'd add that if you provide it something like a slack tool, you can enable the LLM to ask for help (human in the (while) loop).
neumann 1y ago
This is great, and I like seeing all the implementations people are making for themselves. Anyone using any opensource tooling that bundles this effectively to allow different local models to be used in this fashion? I am thinking this would be nice to run fully locally to access my code or my private github repos from my commandline and switch models out (assuming through llama.ccp or Ollama)?
- sagarpatil 1y ago
  All IDE’s support OpenAI compatible endpoint. So you can host whatever model you like locally and use it. Check out Roo Code.
subtlesoftware 1y ago
Link to the full script is broken?
SafeDusk 1y ago
Sharing an agent framework (from scratch) that works very well with just 7 composable tools: read, write, diff, browse, command, think and ask. Just pushed an update this week for OpenAI-compatibility too! https://github.com/aperoc/toolkami https://github.com/aperoc/toolkami
stavros 1y ago
Unfortunately, I haven't had a good experience with tool use. I built a simple agent that can access my calendar and add events, and GPT-4.1 regularly gaslights me, saying "I've added this to your calendar" when it hasn't. It takes a lot of prompting for it to actually add the event, but it will insist it has, even though it never called the tool. Does anyone know of a fix? I'm using the OpenAI agents SDK.
polishdude20 1y ago
I'd love to try this to help me handle a complex off by one cent problem I'm having.
pawanjswal 1y ago
Wild how something so powerful boils down to just 9 lines of code.
lacker 1y ago
I've been using Claude Code, and I really prefer the command line to the IDE-integrated ones. I'm curious about Gemini's increased context size, though. Is anyone successfully using one of the open source CLI agents together with Gemini, and has something to recommend there?
- sagarpatil 1y ago
  Codex CLI with Gemini, yes. It’s not really good at agentic stuff (at least in codex), I’ve had some success with Roo Cline (IDE) but I naturally just end up using Claude Code/Augment Code.
danjc 1y ago
We built tools to give context to an ai chat help embedded in our product. Included is the ability for it to see recent activity logs, the definition of the current object and the ability to search and read help articles. The quality of the chats still amazes me months later. Where we find it got something wrong, we add more detail to the relevant help articles.
bicepjai 1y ago
Which agent is token hungry ? I notice cline is top on the list. Roo eats less than cline. Are there agents that we can configure how the interactions go ? How does Claude code compare to other agents ?
stpedgwdgfhgdd 1y ago
Bit of topic, but worth sharing; Yesterday was for me a milestone, i connected Claude Code through MCP with Jira (sse). I asked it to create a plan for a specific Jira issue, ah, excuse me, work item. CC created the plan based on the item’s description and started coding. It created a branch (wrong naming convention, needs fix), made the code changes and pushed. Since the Jira item had a good description, the plan was solid and the code so far as well. Disclaimer; this was a simple problem to solve, but the code base is pretty large.
suninsight 1y ago
It only seems effective, unless you start using it for actual work. The biggest issue - context. All tool use creates context. Large code bases come with large context out of the bat. LLM's seem to work, unless they are hit with a sizeable context. Anything above 10k and the quality seems to deteriorate. Other issue is that LLM's can go off on a tangent. As context builds up, they forget what their objective was. One wrong turn, and in the rabbit hole they go never to recover. The reason I know, is because we started solving these problems an year back. And we aren't done yet. But we did cover a lot of distance. [Plug]: Try it out at https://nonbios.ai https://nonbios.ai: - Agentic memory → long-horizon coding - Full Linux box → real runtime, not just toy demos - Transparent → see & control every command - Free beta — no invite needed. Works with throwaway email (mailinator etc.)
- toit4wing 1y ago
  Looks interesting. How do you manage context ?
  - suninsight 1y ago
    So managing context is what takes the maximum effort. We use a bunch of strategies to reduce it, including, but not limited to: 1. Custom MCP server to work on linux command line. This wasn't really a 'MCP' server because we started working on it before MCP was a thing. But thats the easiest way to explain it now. The MCP server is optimised to reduce context. 2. Guardrails to reduce context. Think about it as prompt alterations giving the LLM subtle hints to work with less context. The hints could be at a behavioural level and a task level. 3. Continuously pruning the built up context to make the Agent 'forget'. Forgetting what is not important is what we believe a foundational capability. This is kind of inspired by the science which says humans use sleep to 'forget' not useful memories and is critical to keeping the brain healthy. This translates directly to LLM's - making them forget is critical to keep them focussed on the larger task and their actions alligned.
- moffkalast 1y ago
  Some of the thinking models might recover... with an extra 4k tokens used up in <thinking>. And even if they were stable at long contexts, the speed drops massively. You just can't win with this architecture lol.
Quenby 1y ago
[dead]
mtaras 1y ago
Just a couple of days ago I discovered this truth myself while building a proactive personal assistant. It boiled down to just giving it access to managing notes and messaging me, and calling it periodically with chat history and it's notes provided. It's surprisingly intelligent and helpful, even though I'm using model that's far from being SOTA (Gemini Flash 2.5)
- dandaka 1y ago
  Would love to learn more! Managing notes — is it your already existing docs? I have been thinking about proactive assistants, but don't really know where to start. I have a few product ideas around that, where this proactivity can deliver a lot of value. My personal experience with building such agents is kinda frustrating so far. But I was only vibe coding for a small amount of time, maybe I need to invest more.
  - mtaras 1y ago
    No, I personally don't have notes that would be valuable enough for the system I was looking for, so notes is just how I'm calling my agent's long-term memory. All I do is provide it with tools to message me and manage it's own notes, useful context (recent chat history, its saved notes, calendar events, etc), and it can act upon the info. The elegance of the system unfolded when I realized that I can not specify any interaction rules beforehand — I just talk to the system, it saves notes for itself, and later acts upon them. I've only started testing it, but so far it's been working as intended.
    - triangleman 1y ago
      What do you mean by "message me" and "later" in this context?
      - mtaras 1y ago
        "Message me" is through a bot in my main messenger app, so the agent is just a chat window for me. "Later" happens per my requests, and agent has an ability to do something later because I just automatically invoke it on a schedule periodically, and ask it "check notes and chat history — do you need to do something, like message the user to remind them of something", and agent has the message_user tool to do that (that goes to the messenger bot API). Or it can decide to do nothing.
deleted 1y ago
[deleted]
jonstewart 1y ago
Maybe I’m just writing code differently than many people, but I don’t spend much time executing complicated or unique shell commands. I write some code, I run make check (alias “mkc”), I run git add —update (alias “gau”), I review my commit with git diff —cached (“gdc”), and I commit (“gcm”). I can see how an LLM is useful when needing to research which tool arguments to use for a particular situation, but particular situations are infrequent. And based on how frequently wrong coding assistants are with their suggestions, I am leery of letting them run commands against my filesystem. What am I missing?
amunozo 1y ago
Has anyone tried to build something like this with a local model such as Qwen 3?
baalimago 1y ago
I'm just going to shamelessly selfplug the blogpost I wrote about this in August last year: https://lorentz.app/blog-item.html?id=clai&heading=tooling https://lorentz.app/blog-item.html?id=clai&heading=tooling
Koshima 1y ago
It's fascinating how quickly the ecosystem around LLM agents is evolving. I think a big part of this "unreasonable effectiveness" comes from the fact that most of these tools are essentially chaining high-confidence steps together without requiring perfect outputs at each stage. The trick is finding the right balance between autonomy and supervision. I wonder if we'll soon see an "agent stack" emerge, similar to the full-stack frameworks in web development, where different layers handle prompts, memory, tool calls, and state management.
tom_m 1y ago
Definitely going to have some "super worms" out there in the future.
mips_avatar 1y ago
Really good explanation of agent loops!

447 points by crawshaw 1y ago

This is what the no-code API-to-MCP creator uses at usetexture.com! I was surprised to find out this is not what the Claude client uses (as of May 2025)

_bin_ 1y ago

I've found sonnet-3.7 to be incredibly inconsistent. It can do very well but has a strong tendency to get off-track and run off and do weird things. 3.5 is better for this, ime. I hooked claude desktop up to an MCP server to fake claude-code less the extortionate pricing and it works decently. I've been trying to apply it for rust work; it's not great yet (still doesn't really seem to "understand" rust's concepts) but can do some stuff if you make it `cargo check` after each change and stop it if it doesn't. I expect something like o3-high is the best out there (aider leaderboards support this) either alone or in combination with 4.1, but tbh that's out of my price range. And frankly, I can't mentally get past paying a very high price for an LLM response that may or may not be useful; it leaves me incredibly resentful as a customer that your model can fail the task, requiring multiple "re-rolls", and you're passing that marginal cost to me.

agilebyte 1y ago
I am avoiding the cost of API access by using the chat/ui instead, in my case Google Gemini 2.5 Pro with the high token window. Repomix a whole repo. Paste it in with a standard prompt saying "return full source" (it tends to not follow this instruction after a few back and forths) and then apply the result back on top of the repo (vibe coded https://github.com/radekstepan/apply-llm-changes https://github.com/radekstepan/apply-llm-changes to help me with that). Else yeah, $5 spent on Cline with Claude 3.7 and instead of fixing my tests, I end up with if/else statements in the source code to make the tests pass.
- harvey9 1y ago
  Guess it was trained by scraping thedailywtf.com
- actsasbuffoon 1y ago
  I decided to experiment with Claude Code this month. The other day it decided the best way to fix the spec was to add a conditional to the test that causes it to return true before getting to the thing that was actually supposed to be tested. I’m finding it useful for really tedious stuff like doing complex, multi step terminal operations. For the coding… it’s not been great.
  - nico 1y ago
    I’ve had this in different ways many times. Like instead of resolving the underlying issue for an exception, it just suggests catching the exception and keep going It also depends a lot on the mix of model and type of code and libraries involved. Even in different days the models seem to be more or less capable (I’m assuming they get throttled internally - this is very noticeable sometimes in how they try to save on output tokens and summarize the code responses as much as possible, at least in the chat/non-api interfaces)
  - christophilus 1y ago
    Well, that’s proof that it used my GitHub projects in its training data.
- nico 1y ago
  Cool tool. What format does it expect from the model? I’ve been looking for something that can take “bare diffs” (unified diffs without line numbers), from the clipboard and then apply them directly on a buffer (an open file in vscode) None of the paste diff extension for vscode work, as they expect a full unified diff/patch I also tried a google-developed patch tool, but also wasn’t very good at taking in the bare diffs, and def couldn’t do clipboard
  - agilebyte 1y ago
layoric 1y ago
I've been using Mistral Medium 3 last couple of days, and I'm honestly surprised at how good it is. Highly recommend giving it a try if you haven't, especially if you are trying to reduce costs. I've basically switched from Claude to Mistral and honestly prefer it even if costs were equal.
- nico 1y ago
  How are you running the model? Mistral’s api or some local version through ollama, or something else?
  - kyleee 1y ago
    Is mistral on open router?
    - nico 1y ago
      Yup https://openrouter.ai/provider/mistral https://openrouter.ai/provider/mistral I guess it can't really be run locally https://www.reddit.com/r/LocalLLaMA/comments/1kgyfif/introducing_mistral_medium_3/ https://www.reddit.com/r/LocalLLaMA/comments/1kgyfif/introdu...
  - layoric 1y ago
    Through OpenRouter, medium 3 isn't open weights.
johnsmith1840 1y ago
I seem to be alone in this but the only methods truly good at coding are slow heavy test time compute models. o1-pro and o1-preview are the only models I've ever used that can reliably update and work with 1000 LOC without error. I don't let o3 write any code unless it's very small. Any "cheap" model will hallucinate or fail massively when pushed. One good tip I've done lately. Remove all comments in your code before passing or using LLMs, don't let LLM generated comments persist under any circumstance.
- _bin_ 1y ago
  Interesting. I've never tested o1-pro because it's insanely expensive but preview seemed to do okay. I wouldn't be shocked if huge, expensive-to-run models performed better and if all the "optimized" versions were actually labs trying to ram cheaper bullshit down everyone's throat. Basically chinesium for LLMs; you can afford them but it's not worth it. I remember someone saying o1 was, what, 200B dense? I might be misremembering.
  - johnsmith1840 1y ago
    I'm positive they are pushing users to cheaper models due to cost. o1-pro is now in a sub menu for pro users and labled legacy. The big inference methods must be stupidly expensive. o1-preview was and possibly still is the most powerful model they ever released. I only switched to pro for coding after months of them improving it and my api bill getting a bit crazy (like 0.50$ per question). I don't think paramater count matters anymore. I think the only thing that matters is how much compute a vendor will give you per question.
- doug_durham 1y ago
  I never have LLMs work on 1000 LOC. I don't think that's the value-add. Instead I use it a the function and class level to accelerate my work. The thought of having any agent human or computer run amok in my code makes me uncomfortable. At the end of the day I'm still accountable for the work, and I have to read and comprehend everything. If do it piecewise I it makes tracking the work easier.

jbellis 1y ago

Yes! Han Xiao at Jina wrote a great article that goes into a lot more detail on how to turn this into a production quality agentic search: https://jina.ai/news/a-practical-guide-to-implementing-deepsearch-deepresearch/ https://jina.ai/news/a-practical-guide-to-implementing-deeps... This is the same principle that we use at Brokk for Search and for Architect. (https://brokk.ai/ https://brokk.ai/) The biggest caveat: some models just suck at tool calling, even "smart" models like o3. I only really recommend Gemini Pro 2.5 for Architect (smart + good tool calls); Search doesn't require as high a degree of intelligence and lots of models work (Sonnet 3.7, gpt-4.1, Grok 3 are all fine).

crawshaw 1y ago
I'm curious about your experiences with Gemini Pro 2.5 tool calling. I have tried using it in agent loops (in fact, sketch has some rudimentary support I added), and compared with the Anthropic models I have had to actively reprompt Gemini regularly to make tool calls. Do you consider it equivalent to Sonnet 3.7? Has it required some prompt engineering?
- jbellis 1y ago
  Confession time: litellm still doesn't support parallel tool calls with Gemini models [https://github.com/BerriAI/litellm/issues/9686 https://github.com/BerriAI/litellm/issues/9686] so we wrote our own "parallel tool calls" on top of Structured Output. It did take a few iterations on the prompt design but all of it was "yeah I can see why that was ambiguous" kinds of things, no real complaints. GP2.5 does have a different flavor than S3.7 but it's hard to say that one is better or worse than the other [edit: at tool calling -- GP2.5 is definitely smarter in general]. GP2.5 is I would say a bit more aggressive at doing "speculative" tool execution in parallel with the architect, e.g. spawning multiple search agent calls at the same time, which for Brokk is generally a good thing but I could see use cases where you'd want to dial that back.
sagarpatil 1y ago
“Claude Code, better than Sourcegraph, better than Augment Code.” That’s a pretty bold claim, how come you are not at the top of this list then? https://www.swebench.com/ https://www.swebench.com/ “Use frontier models like o3, Gemini Pro 2.5, Sonnet 3.7” Is this unlimited usage? Or number of messages/tokens? Why do you need a separate desktop app? Why not CLI or VS Code extension.

kgeist 1y ago

Today I tried "vibe-coding" for the first time using GPT-4o and 4.1. I did it manually - just feeding compilation errors, warnings, and suggestions in a loop via the canvas interface. The file was small, around 150 lines. It didn't go well. I started with 4o: - It used a deprecated package. - After I pointed that out, it didn't update all usages - so I had to fix them manually. - When I suggested a small logic change, it completely broke the syntax (we're talking "foo() } return )))" kind of broken) and never recovered. I gave it the raw compilation errors over and over again, but it didn't even register the syntax was off - just rewrote random parts of the code instead. - Then I thought, "maybe 4.1 will be better at coding" (as advertized). But 4.1 refused to use the canvas at all. It just explained what I could change - as in, you go make the edits. - After some pushing, I got it to use the canvas and return the full code. Except it didn't - it gave me a truncated version of the code with comments like "// omitted for brevity". That's when I gave up. Do agents somehow fix this? Because as it stands, the experience feels completely broken. I can't imagine giving this access to bash, sounds way too dangerous.

skeeter2020 1y ago
I had an even better experience. I asked to produce a small web app with a new-to-me framework: success! I asked to make some CSS changes to the UI; the app no longer builds.
vFunct 1y ago
Use Claude Sonnet with an IDE.
85392_school 1y ago
Agents definitely fix this. When you can run commands and edit files, the agent can test its code by itself and fix any issues.
visarga 1y ago
You should try Cursor or Windsurf, with Claude or Gemini model. Create a documentation file first. Generate tests for everything. The more the better. Then let it cycle 100 times until tests pass. Normal programming is like walking, deliberate and sure. Vibe coding is like surfing, you can't control everything, just hit yes on auto. Trust the process, let it make mistakes and recover on its own.

libraryofbabel 1y ago

Strongly recommend this blog post too which is a much more detailed and persuasive version of the same point. The author actually goes and builds a coding agent from zero: https://ampcode.com/how-to-build-an-agent https://ampcode.com/how-to-build-an-agent It is indeed astonishing how well a loop with an LLM that can call tools works for all kinds of tasks now. Yes, sometimes they go off the rails, there is the problem of getting that last 10% of reliability, etc. etc., but if you're not at least a little bit amazed then I urge you go to and hack together something like this yourself, which will take you about 30 minutes. It's possible to have a sense of wonder about these things without giving up your healthy skepticism of whether AI is actually going to be effective for this or that use case. This "unreasonable effectiveness" of putting the LLM in a loop also accounts for the enormous proliferation of coding agents out there now: Claude Code, Windsurf, Cursor, Cline, Copilot, Aider, Codex... and a ton of also-rans; as one HN poster put it the other day, it seems like everyone and their mother is writing one. The reason is that there is no secret sauce and 95% of the magic is in the LLM itself and how it's been fine-tuned to do tool calls. One of the lead developers of Claude Code candidly admits this in a recent interview.[0] Of course, a ton of work goes into making these tools work well, but ultimately they all have the same simple core. [0] https://www.youtube.com/watch?v=zDmW5hJPsvQ https://www.youtube.com/watch?v=zDmW5hJPsvQ

sesm 1y ago
Should we change the link above to use `?utm_source=hn&utm_medium=browser` before opening it?
- libraryofbabel 1y ago
  fixed :)
kcorbitt 1y ago
For "that last 10% of reliability" RL is actually working pretty well right now too! https://openpipe.ai/blog/art-e-mail-agent https://openpipe.ai/blog/art-e-mail-agent
meander_water 1y ago

bhouston 1y ago

I found this out too - it is quite easy and effective: https://benhouston3d.com/blog/building-an-agentic-code-from-scratch https://benhouston3d.com/blog/building-an-agentic-code-from-...

simonw 1y ago

I'm very excited about tool use for LLMs at the moment. The trick isn't new - I first encountered it with the ReAcT paper two years ago - https://til.simonwillison.net/llms/python-react-pattern https://til.simonwillison.net/llms/python-react-pattern - and it's since been used for ChatGPT plugins, and recently for MCP, and all of the models have been trained with tool use / function calls in mind. What's interesting today is how GOOD the models have got at it. o3/o4-mini's amazing search performance is all down to tool calling. Even Qwen3 4B (2.6GB from Ollama, runs happily on my Mac) can do tool calling reasonably well now. I gave a workshop at PyCon US yesterday about building software on top of LLMs - https://simonwillison.net/2025/May/15/building-on-llms/ https://simonwillison.net/2025/May/15/building-on-llms/ - and used that as an excuse to finally add tool usage to an alpha version of my LLM command-line tool. Here's the section of the workshop that covered that: https://building-with-llms-pycon-2025.readthedocs.io/en/latest/tools.html https://building-with-llms-pycon-2025.readthedocs.io/en/late... My LLM package can now reliably count the Rs in strawberry as a shell one-liner: llm --functions ' def count_char_in_string(char: str, string: str) -> int: """Count the number of times a character appears in a string.""" return string.lower().count(char.lower()) ' 'Count the number of Rs in the word strawberry' --td

andrewmcwatters 1y ago
I love the odd combination of silliness and power in this.
DarmokJalad1701 1y ago
Was the workshop recorded?
- simonw 1y ago
  No video or audio, just my handouts.

maxqper123 1y ago

[flagged]

mukesh610 1y ago

I built this very same thing today! The only difference is that i pushed the tool call outputs into the conversation history and resent it back to the LLM for it to summarize, or perform further tool calls, if necessary, automagically. I used ollama to build this and ollama supports tool calling natively, by passing a `tools=[...]` in the Python SDK. The tools can be regular Python functions with docstrings that describe the tool use. The SDK handles converting the docstrings into a format the LLM can recognize, so my tool's code documentation becomes the model's source of truth. I can also include usage examples right in the docstring to guide the LLM to work closely with all my available tools. No system prompt needed! Moreover, I wrote all my tools in a separate module, and just use `inspect.getmembers` to construct the `tools` list that i pass to Ollama. So when I need to write a new tool, I just write another function in the tools module and it Just Works™ Paired with qwen 32b running locally, i was fairly satisfied with the output.

degamad 1y ago
> The only difference is that i pushed the tool call outputs into the conversation history and resent it back to the LLM for it to summarize, or perform further tool calls, if necessary, automagically. It looks like this one does that too. msg = [ handle_tool_call(tc) for tc in tool_calls ]
- mukesh610 1y ago
  Ah, failed to notice that. I was so excited because this was exactly what I coded up today, I jumped straight to the comments.

jawns 1y ago

Not only can this be an effective strategy for coding tasks, but it can also be used for data querying. Picture a text-to-SQL agent that can query database schemas, construct queries, run explain plans, inspect the error outputs, and then loop several times to refine. That's the basic architecture behind a tool I built, and I have been amazed at how well it works. There have been multiple times when I've thought, "Surely it couldn't handle THIS prompt," but it does! Here's an AWS post that goes into detail about this approach: https://aws.amazon.com/blogs/machine-learning/build-a-robust-text-to-sql-solution-generating-complex-queries-self-correcting-and-querying-diverse-data-sources/ https://aws.amazon.com/blogs/machine-learning/build-a-robust...

amelius 1y ago

Huh, isn't this already built-in, in most chat UIs?

bkyan 1y ago
This is for running locally in the terminal (or in WSL2 if you are on Windows). Here is an example session from a similar script that I <cough>ChatGPT</cough> wrote in Perl, illustrating the same concept: $ ~/agentloop.pl YOUR PROMPT: Is Python installed on this system? If so, which version? BASH STDIN: python --version 2>&1 || python3 --version 2>&1 BASH STDOUT: /bin/bash: python: command not found Python 3.6.7 LLM RESPONSE: Python is installed on this system. The version available is Python 3.6.7. However, it looks like the python executable (python) is not directly available, but python3 (or a direct version path) exists and resolves to Python 3.6.7. If you need to use Python, you should use python3 as the command.
cheema33 1y ago
Which chat UI lets me use my local tools? Like git, find, pnpm, curl...?
- insin 1y ago
  Has anybody written an Electron app yet which injects tools into existing chat UIs, letting you you expose whatever you want from your machine to them via Node? I was planning to create something like that for the BigCo internal chat UI I work on.
  - energy123 1y ago
    Someone wrote a browser extension that does that for Gemini web chat
  - NitpickLawyer 1y ago
    Careful, this might break the ToS on some products. At least for those that offer all you can prompt in a browser vs. API.

BrandiATMuhkuh 1y ago

That's really cool. One week ago I implemented an SQL tool and it works really well. But sometimes it still just makes up table/column names. Luckily it can read the error and correct itself. But today I went to the next level. I gave the LLM two tools. One web search tool and one REST tool. I told it at what URL it can find API docs. Then I asked it to perform some tasks for me. It was really cool to watch an AI read docs, make api calls and try again (REPL) until it worked

rbren 1y ago

If you're interested in hacking on agent loops, come join us in the OpenHands community! Here's our (slightly more complicated) agent loop: https://github.com/All-Hands-AI/OpenHands/blob/f7cb2d0f64666e1f090a5152d7c002aa6f28caf9/openhands/controller/agent_controller.py#L771 https://github.com/All-Hands-AI/OpenHands/blob/f7cb2d0f64666...

sagarpatil 1y ago
I tried OH yesterday. Sorry to say this but your UI/UX experience is horrible. I don’t understand why do you need a separate UI instead of using local IDE (Cursor/Windsurf), vs code extension (augment) or CLI (Amazon Q developer). Please do not reinvent the wheel.

tqwhite 1y ago

I've been using Claude Code, ie, a terminal interface to Sonnet 3.7 since the day it came out in mid March. I have done substantial CLI apps, full stack web systems and a ton of utility crap. I am much more ambitious because of it, much as I was in the past when I was running a programming team. I'm sure it is much the same as this under the hood though Anthropic has added many insanely useful features. Nothing is perfect. Producing good code requires about the same effort as it did when I was running said team. It is possible to get complicated things working and find oneself in a mess where adding the next feature is really problematic. As I have learned to drive it, I have to do much less remediation and refactoring. That will never go away. I cannot imagine what happened to poor kgeist. I have had Claude make choices I wouldn't and do some stupid stuff, never enough that I would even think about giving up on it. Almost always, it does a decent job and, for a most stuff, the amount of work it takes off of my brain is IMMENSE. And, for good measure, it does a wonderful job of refactoring. Periodically, I have a session where I look at the code, decide how it could be better and instruct Claude. Huge amounts of complexity, done. "Change this data structure", done. It's amazingly cool. And, just for fun, I opened it in a non-code archive directory. It was a junk drawer that I've been filling for thirty years. "What's in this directory?" "Read the old resumes and write a new one." "What are my children's names?" Also amazing. And this is still early days. I am so happy.

benoau 1y ago
> And, for good measure, it does a wonderful job of refactoring. Periodically, I have a session where I look at the code, decide how it could be better and instruct Claude. Huge amounts of complexity, done. "Change this data structure", done. It's amazingly cool. Yeah this is literally just so enjoyable. Stuff that would be an up-hill battle to get included in a sprint takes 5 minutes. It makes it feel like a whole team is just sitting there, waiting to eagerly do my bidding with none of the headache waiting for work to be justified, scheduled, scoped, done, and don't even have to justify rejecting it if I don't like the results.
felipeerias 1y ago
Recently I had to define a remote data structure, specify the API to request it, implement parsing and storage, and show it to the user. Claude was able to handle all of these tasks simultaneously, so I could see how small changes at either end would impact the intermediate layers. I iterated on many ideas until I settled on the best overall solution for my use case. Being able to iterate like that through several layers of complexity was eye-opening. It made me more productive while giving me a better understanding of how the different components fit together.

kuahyeow 1y ago

What protection do people use when enabling an LLM to run `bash` on your machine ? Do you run it in a Docker container / LXC boundary ? `chroot` ?

CGamesPlay 1y ago
The blog post in question is on the site for Sketch, which appears to use Docker containers. That said, I use Claude Code, which just uses unsandboxed commands with manual approval. What's your concern? An accident or an attacker? For accidents, I use git and backups and develop in a devcontainer. For an attacker, bash just seems like an ineffective attack vector; I would be more worried about instructing the agent to write a reverse shell directly into the code.
- zahlman 1y ago
  > For an attacker, bash just seems like an ineffective attack vector All it needs to do is curl and run the actual payload.
  - deleted 1y ago
    [deleted]
  - Wilder7977 1y ago
    Or even the standard "echo xxx | base64 -d" or a million other ways. How can someone say that bash is not interesting to an attacker is beyond me.
    - CGamesPlay 1y ago
      But bash isn't a key ingredient in any of these. The exact same payload could easily be insert in the project's source code, and has the benefit of being persistent. Using a bash shell to do it might be the most obvious way, sure, but shutting down bash access is such a poor defense that it isn't worth doing.
- cess11 1y ago

outworlder 1y ago

> If you don't have some tool installed, it'll install it. Terrifying. LLMs are very 'accommodating' and all they need is someone asking them to do something. This is like SQL injection, but worse.

rglover 1y ago
I often wonder what the first agent-driven catastrophe will be. Considering the gold rush (emphasis on rush) going on, it's only a matter of time before a difficult to fix disaster occurs.
nxobject 1y ago
In an ideal world, this would require us as programmers to lean into our codebase reading and augmentation skills – currently underappreciated anyway as a skill to build. But when the incentives lean towards write-only code, I'm not optimistic.

cadamsdotcom 1y ago

> "Oh, this test doesn't pass... let's just skip it," it sometimes says, maddeningly. Here is a wild idea. Imagine running a companion, policy-enforcing LLM, independently and in parallel, which is given instructions to keep the main LLM behaving according to instructions. If the companion LLM could - in real time - ban the coding LLM from emitting "let's just skip it" by seeing the tokens "let's just" and then biasing the output such that the word "skip" becomes impossible to emit. Banning the word "skip" from following "let's just", forces the LLM down a new path away from the undesired behavior. It's like Structured Outputs or JSON mode, but driven by a companion LLM, and dynamically modified in real time as tokens are emitted. If the idea works, you could prompt the companion LLM to do more advanced stuff - eg. ban a coding LLM from making tests pass by deleting the test code, ban it from emitting pointless comments... all the policies that we put into system prompts today and pray the LLM will do, would go into the companion LLM's prompt instead. Wonder what the Outlines folks think of this!

JoshuaDavid 1y ago
Along these lines, if the main LLM goes down a bad path, you could _rewind_ the model to before it started going down the bad path -- the watcher LLM doesn't necessarily have to guess that "skip" is a bad token after the words "let's just", it could instead see "let's just skip the test" and go "nope, rolling back to the token "just " and rerolling with logit_bias={"skip":-10,"omit":-10,"hack":-10}". Of course doing that limits which model providers you can work with (notably, OpenAI has gotten quite hostile to power users doing stuff like that over the past year or so).
- cadamsdotcom 1y ago
  That’s a really neat idea. Kind of seems an optimization: if the “token ban” is a tool call, you can see that being too slow to run for every token. Provided rewinding is feasible, your idea could make it performant enough to be practical.
  - TeMPOraL 1y ago
    It's not an optimization; the greedy approach won't work well, because it rejects critical context that comes after the tokens that would trigger the rejection. Consider: "Let's just skip writing this test, as the other change you requested seems to eliminate the need for it." Rolling back the model on "Let's just" would be stupid; rolling it on "Let's just skip writing this test" would be stupid too, as is the beliefs that writing tests is your holy obligation to your god, and you must do so unconditionally. The following context makes it clear that the decision is fine. Or, if you (or the governor agent) don't buy the reasoning, you're then in a perfect position to say, "nope, let's roll back to <Let's> and bias against ["skip, test"]". Checking the entire message once makes sense; checking it after every token doesn't.

magicalhippo 1y ago

Assuming the title is a play on the paper "The Unreasonable Effectiveness of Mathematics in the Natural Sciences"[1][2] by Eugene Wigner. [1]: https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness... [2]: https://www.hep.upenn.edu/~johnda/Papers/wignerUnreasonableEffectiveness.pdf https://www.hep.upenn.edu/~johnda/Papers/wignerUnreasonableE...

gavmor 1y ago
That may be its primogenitor, but it's long since become a meme: https://scholar.google.com/scholar?q=unreasonable+effectiveness https://scholar.google.com/scholar?q=unreasonable+effectiven...
- -__---____-ZXyw 1y ago
  a. Primogenitor is a nice word! b. Wigner's original essay is a serious piece, and quite beautiful in its arguments. I had been under the impression that the phrasing had been used a few times since, but typically by other serious people who were aware of the lineage of that lovely essay. With this 6-paragraph vibey-blog-post, it truly has become a meme. So it goes, I suppose.
dsubburam 1y ago
I didn't know of that paper, and thought the title was a riff on Karpathy's Unreasonable Effectiveness of RNNs in 2015[1]. Even if my thinking is correct, as it very well might be given the connection RNNs->LLMs, Karpathy might have himself made his title a play on Wigner's (though he doesn't say so). [1] https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://karpathy.github.io/2015/05/21/rnn-effectiveness/
- throwaway314155 1y ago
  Unreasonable effectiveness of [blah] has been a thing for decades if not centuries. It's not new.

leelou2 1y ago

[dead]

benoau 1y ago

This morning I used cursor to extract a few complex parts of my game prototype's "main loop", and then generate a suite of tests for those parts. In total I have 341 tests written by Cursor covering all the core math and other components. It has been a bit like herding cats sometimes, it will run away with a bad idea real fast, but the more constraints I give it telling it what to use, where to put it, giving it a file for a template, telling it what not to do, the better the results I get. In total it's given me 3500 lines of test code that I didn't need to write, don't need to fix, and can delete and regenerate if underlying assumptions change. It's also helped tune difficulty curves, generate mission variations and more.

otterley 1y ago
Writing tests, in my experience, is by far the best use case for LLMs. It eliminates hours or days of drudgery and toil, and can provide coverage of so many more edge cases that I can think of myself. Plus it makes your code more robust! It’s just wonderful all around.
- benoau 1y ago
  Yeah I generated another 60 tests / 500 lines of test code since I posted my comment. I must have written 100,000s of lines of test codes the hard way, it's not just faster than me it's better than me when it comes to the coverage, and it requires less of my help than a number of developers I've had under me.

hbbio 1y ago

Yes, agent loops are simple, except, as the article says, a bit of "pump and circumstance"! If anyone is interested, I tried to put together a minimal library (no dependency) for TypeScript: https://github.com/hbbio/nanoagent https://github.com/hbbio/nanoagent

bdbenton5255 1y ago

Woke up this morning to start on a new project. Started with a math visualizer for machine learning, saw an HN post for this soon after and scrapped it. It was better done by someone else. Started on an LLM app that looped outputs, saw this post soon after and scrapped it. It was better done by someone else. It is like every single original notion I have is immediately done by someone else at the exact same time. I think I will just move on to rudimentary systems programming stuff and avoid creative and original thinking, just need basic and low profile employment.

soulofmischief 1y ago
What is your motive for creating things? Does it really matter if there is competition?
throwaway314155 1y ago
> Started on an LLM app that looped outputs, saw this post soon after and scrapped it. It was better done by someone else. If it helps, "TFA" was not the originator here and is merely simplifying concepts from fairly established implementations in the wild. As simonw mentions elsewhere, it goes back to at least the ReAct paper and maybe even more if you consider things like retrieval-augmented generation.
morkalork 1y ago
Everything has been done before, sometimes before you were even born. The trick is to just not google it first and do it anyways.
- Lionga 1y ago
  No. DO google it, learn from it and make it better

themichaellai 1y ago

I've also been defining agents as "LLM call in a while loop with tools" to my co-workers as well — I'd add that if you provide it something like a slack tool, you can enable the LLM to ask for help (human in the (while) loop).

neumann 1y ago

This is great, and I like seeing all the implementations people are making for themselves. Anyone using any opensource tooling that bundles this effectively to allow different local models to be used in this fashion? I am thinking this would be nice to run fully locally to access my code or my private github repos from my commandline and switch models out (assuming through llama.ccp or Ollama)?

sagarpatil 1y ago
All IDE’s support OpenAI compatible endpoint. So you can host whatever model you like locally and use it. Check out Roo Code.

subtlesoftware 1y ago

Link to the full script is broken?

SafeDusk 1y ago

Sharing an agent framework (from scratch) that works very well with just 7 composable tools: read, write, diff, browse, command, think and ask. Just pushed an update this week for OpenAI-compatibility too! https://github.com/aperoc/toolkami https://github.com/aperoc/toolkami

stavros 1y ago

Unfortunately, I haven't had a good experience with tool use. I built a simple agent that can access my calendar and add events, and GPT-4.1 regularly gaslights me, saying "I've added this to your calendar" when it hasn't. It takes a lot of prompting for it to actually add the event, but it will insist it has, even though it never called the tool. Does anyone know of a fix? I'm using the OpenAI agents SDK.

polishdude20 1y ago

I'd love to try this to help me handle a complex off by one cent problem I'm having.

pawanjswal 1y ago

Wild how something so powerful boils down to just 9 lines of code.

lacker 1y ago

I've been using Claude Code, and I really prefer the command line to the IDE-integrated ones. I'm curious about Gemini's increased context size, though. Is anyone successfully using one of the open source CLI agents together with Gemini, and has something to recommend there?

sagarpatil 1y ago
Codex CLI with Gemini, yes. It’s not really good at agentic stuff (at least in codex), I’ve had some success with Roo Cline (IDE) but I naturally just end up using Claude Code/Augment Code.

danjc 1y ago

We built tools to give context to an ai chat help embedded in our product. Included is the ability for it to see recent activity logs, the definition of the current object and the ability to search and read help articles. The quality of the chats still amazes me months later. Where we find it got something wrong, we add more detail to the relevant help articles.

bicepjai 1y ago

Which agent is token hungry ? I notice cline is top on the list. Roo eats less than cline. Are there agents that we can configure how the interactions go ? How does Claude code compare to other agents ?

stpedgwdgfhgdd 1y ago

Bit of topic, but worth sharing; Yesterday was for me a milestone, i connected Claude Code through MCP with Jira (sse). I asked it to create a plan for a specific Jira issue, ah, excuse me, work item. CC created the plan based on the item’s description and started coding. It created a branch (wrong naming convention, needs fix), made the code changes and pushed. Since the Jira item had a good description, the plan was solid and the code so far as well. Disclaimer; this was a simple problem to solve, but the code base is pretty large.

suninsight 1y ago

It only seems effective, unless you start using it for actual work. The biggest issue - context. All tool use creates context. Large code bases come with large context out of the bat. LLM's seem to work, unless they are hit with a sizeable context. Anything above 10k and the quality seems to deteriorate. Other issue is that LLM's can go off on a tangent. As context builds up, they forget what their objective was. One wrong turn, and in the rabbit hole they go never to recover. The reason I know, is because we started solving these problems an year back. And we aren't done yet. But we did cover a lot of distance. [Plug]: Try it out at https://nonbios.ai https://nonbios.ai: - Agentic memory → long-horizon coding - Full Linux box → real runtime, not just toy demos - Transparent → see & control every command - Free beta — no invite needed. Works with throwaway email (mailinator etc.)

toit4wing 1y ago
Looks interesting. How do you manage context ?
- suninsight 1y ago
  So managing context is what takes the maximum effort. We use a bunch of strategies to reduce it, including, but not limited to: 1. Custom MCP server to work on linux command line. This wasn't really a 'MCP' server because we started working on it before MCP was a thing. But thats the easiest way to explain it now. The MCP server is optimised to reduce context. 2. Guardrails to reduce context. Think about it as prompt alterations giving the LLM subtle hints to work with less context. The hints could be at a behavioural level and a task level. 3. Continuously pruning the built up context to make the Agent 'forget'. Forgetting what is not important is what we believe a foundational capability. This is kind of inspired by the science which says humans use sleep to 'forget' not useful memories and is critical to keeping the brain healthy. This translates directly to LLM's - making them forget is critical to keep them focussed on the larger task and their actions alligned.
moffkalast 1y ago
Some of the thinking models might recover... with an extra 4k tokens used up in <thinking>. And even if they were stable at long contexts, the speed drops massively. You just can't win with this architecture lol.

Quenby 1y ago

[dead]

mtaras 1y ago

Just a couple of days ago I discovered this truth myself while building a proactive personal assistant. It boiled down to just giving it access to managing notes and messaging me, and calling it periodically with chat history and it's notes provided. It's surprisingly intelligent and helpful, even though I'm using model that's far from being SOTA (Gemini Flash 2.5)

dandaka 1y ago
Would love to learn more! Managing notes — is it your already existing docs? I have been thinking about proactive assistants, but don't really know where to start. I have a few product ideas around that, where this proactivity can deliver a lot of value. My personal experience with building such agents is kinda frustrating so far. But I was only vibe coding for a small amount of time, maybe I need to invest more.
- mtaras 1y ago
  No, I personally don't have notes that would be valuable enough for the system I was looking for, so notes is just how I'm calling my agent's long-term memory. All I do is provide it with tools to message me and manage it's own notes, useful context (recent chat history, its saved notes, calendar events, etc), and it can act upon the info. The elegance of the system unfolded when I realized that I can not specify any interaction rules beforehand — I just talk to the system, it saves notes for itself, and later acts upon them. I've only started testing it, but so far it's been working as intended.
  - triangleman 1y ago
    What do you mean by "message me" and "later" in this context?
    - mtaras 1y ago
      "Message me" is through a bot in my main messenger app, so the agent is just a chat window for me. "Later" happens per my requests, and agent has an ability to do something later because I just automatically invoke it on a schedule periodically, and ask it "check notes and chat history — do you need to do something, like message the user to remind them of something", and agent has the message_user tool to do that (that goes to the messenger bot API). Or it can decide to do nothing.

deleted 1y ago

[deleted]

jonstewart 1y ago

Maybe I’m just writing code differently than many people, but I don’t spend much time executing complicated or unique shell commands. I write some code, I run make check (alias “mkc”), I run git add —update (alias “gau”), I review my commit with git diff —cached (“gdc”), and I commit (“gcm”). I can see how an LLM is useful when needing to research which tool arguments to use for a particular situation, but particular situations are infrequent. And based on how frequently wrong coding assistants are with their suggestions, I am leery of letting them run commands against my filesystem. What am I missing?

amunozo 1y ago

Has anyone tried to build something like this with a local model such as Qwen 3?

baalimago 1y ago

I'm just going to shamelessly selfplug the blogpost I wrote about this in August last year: https://lorentz.app/blog-item.html?id=clai&heading=tooling https://lorentz.app/blog-item.html?id=clai&heading=tooling

Koshima 1y ago

It's fascinating how quickly the ecosystem around LLM agents is evolving. I think a big part of this "unreasonable effectiveness" comes from the fact that most of these tools are essentially chaining high-confidence steps together without requiring perfect outputs at each stage. The trick is finding the right balance between autonomy and supervision. I wonder if we'll soon see an "agent stack" emerge, similar to the full-stack frameworks in web development, where different layers handle prompts, memory, tool calls, and state management.

tom_m 1y ago

Definitely going to have some "super worms" out there in the future.

mips_avatar 1y ago

Really good explanation of agent loops!