7 ms·
Strongly recommend this blog post too which is a much more detailed and persuasive version of the same point. The author actually goes and builds a coding agent
by libraryofbabel 1y ago
Strongly recommend this blog post too which is a much more detailed and persuasive version of the same point. The author actually goes and builds a coding agent from zero: https://ampcode.com/how-to-build-an-agent https://ampcode.com/how-to-build-an-agent
It is indeed astonishing how well a loop with an LLM that can call tools works for all kinds of tasks now. Yes, sometimes they go off the rails, there is the problem of getting that last 10% of reliability, etc. etc., but if you're not at least a little bit amazed then I urge you go to and hack together something like this yourself, which will take you about 30 minutes. It's possible to have a sense of wonder about these things without giving up your healthy skepticism of whether AI is actually going to be effective for this or that use case.
This "unreasonable effectiveness" of putting the LLM in a loop also accounts for the enormous proliferation of coding agents out there now: Claude Code, Windsurf, Cursor, Cline, Copilot, Aider, Codex... and a ton of also-rans; as one HN poster put it the other day, it seems like everyone and their mother is writing one. The reason is that there is no secret sauce and 95% of the magic is in the LLM itself and how it's been fine-tuned to do tool calls. One of the lead developers of Claude Code candidly admits this in a recent interview.[0] Of course, a ton of work goes into making these tools work well, but ultimately they all have the same simple core.
[0] https://www.youtube.com/watch?v=zDmW5hJPsvQ https://www.youtube.com/watch?v=zDmW5hJPsvQ
- sesm 1y agoShould we change the link above to use `?utm_source=hn&utm_medium=browser` before opening it?
- libraryofbabel 1y agofixed :)
- kcorbitt 1y agoFor "that last 10% of reliability" RL is actually working pretty well right now too! https://openpipe.ai/blog/art-e-mail-agent https://openpipe.ai/blog/art-e-mail-agent
- meander_water 1y agoThere's also this one which uses pocketflow, a graph abstraction library to create something similar [0]. I've been using it myself and love the simplicity of it. [0] https://github.com/The-Pocket/PocketFlow-Tutorial-Cursor/blob/main/blog.md https://github.com/The-Pocket/PocketFlow-Tutorial-Cursor/blo...
- wepple 1y agoAh, it’s Thorsten Ball! I thoroughly enjoyed his “writing an interpreter”. I guess I’m going to build an agent now.
- deleted 1y ago[deleted]
- aibrother 1y agothanks for the rec. and yeah agreed with the observations as well
- datpuz 1y agoCan't think of anything an LLM is good enough at to let them do on their own in a loop for more than a few iterations before I need to reign it back in.
- CuriouslyC 1y agoThe main problem with agents is that they aren't reflecting on their own performance and pausing their own execution to ask a human for help aggressively enough. Agents can run on for 20+ iterations in many cases successfully, but also will need hand holding after every iteration in some cases. They're a lot like a human in that regard, but we haven't been building that reflection and self awareness into them so far, so it's like a junior that doesn't realize when they're over their depth and should get help.
- ariwilson 1y agoIs there value in adding an overseer LLM that measures the progress between n steps and if it's too low stops and calls out to a human?
- solumunus 1y agoAnd how does it effectively measure progress?
- NotMichaelBay 1y agoIt can behave just like a senior role would - produce the set of steps for the junior to follow, and assess if the junior appears stuck at any particular step.
- chongli 1y agoProducing the set of steps is the hard part. If you can do that, you don’t need a junior to follow it, you have a program to execute.
- orange_puff 1y agoI have been trying to find such an article for so long, thank you! I think a common reaction to Agents is “well, it probably cannot solve a really complex problem very well”. But to me, that isn’t the point of an agent. LLMs function really well with a lot of context, and agent allows the LLM to discover more context and improve its ability to answer questions.
- deadbabe 1y agoGenerally when LLM’s are effective like this, it means a more efficient non-LLM based solution to the problem exists using the tools you have provided. The LLM helps you find the series of steps and synthesis of inputs and outputs to make it happen. It is expensive and slow to have an LLM use tools all the time for solving the problem. The next step is to convert frequent patterns of tool calls into a single pure function, performing whatever transformation of inputs and outputs are needed along the way (an LLM can help you build these functions), and then perhaps train a simple cheap classifier to always send incoming data to this new function, bypassing LLMs all together. In time, this will mean you will use LLMs less and less, limiting their use to new problems that are unable to be classified. This is basically like a “cache” for LLM based problem solving, where the keys are shapes of problems. The idea of LLMs running 24/7 solving the same problems in the same way over and over again should become a distant memory, though not one that an AI company with vested interest in selling as many API calls as possible will want people to envision. Ideally LLMs are only needed to be employed once or a few times per novel problem before being replaced with cheaper code.
- toobulkeh 1y agoI’ve been tinkering with this, but haven’t found a pattern or library of someone solving this. Have you?
- vidarh 1y agoThere's a Ruby port of the first article you linked as well. Feature-wise they're about the same, but if you (like me) enjoy Ruby more than Python it's worth reading both articles: https://news.ycombinator.com/item?id=43984860 https://news.ycombinator.com/item?id=43984860 https://radanskoric.com/articles/coding-agent-in-ruby https://radanskoric.com/articles/coding-agent-in-ruby
- forgingahead 1y agoLove to see the Ruby implementations! Thanks for sharing.
- ichiwells 1y agoThank you so much for sharing this! We are using ruby to build a powerful AI toolset in the construction space, and we love how simple all of the SaaS parts are and not reinventing the wheel, but the ruby LLM SDK ecosystem is a bit lagging, so we've written a lot of our own low-level tools. (btw we are also hiring rubyists https://news.ycombinator.com/item?id=43865448 https://news.ycombinator.com/item?id=43865448)
- vidarh 1y agoGreat to see someone pushing Ruby in this space. Not after anything permanent, but if you're ever after "plugging gaps" with contract work, complex Ruby projects is what I enjoy the most.
- gchamonlive 1y agoHow far can you go with the best models that fit in a consumer grade GPU (24GB vram)?
- xnx 1y ago> The reason is that there is no secret sauce and 95% of the magic is in the LLM itself Makes that "$3 billion" valuation for Windsurf very suspect
- rrrx3 1y agoIndeed. But keep in mind they weren't just buying the tooling - they get the team, the brand, and the positional authority as well. OpenAI could have spun up a team to build an agent code IDE, and they would have been starting on the back foot with users, would have been compared to Cursor/Windsurf... The price tag is hefty but I figure it'll work out for them on the backside because they won't have to fight so hard to capture TAM.
- TonyEx 1y agoThe value in the windsurf acquisition isn't the code they've written, it's the ability to see what people are coding and use that information to build better LLMs. -- Product development.
- TZubiri 1y agothere is the problem of getting that last 10% of reliability. In my experience, that next 9% will take 9 times the effort. And that next 0.9% will take 9 times the effort. And so on. So 90% is very far off from 99.999% reliability. Which would still be less reliable than an ec2 instance.
- com 1y agoAs the project management saying goes: 99% done is much closer to 0% done than 100% done