10 ms·

Agent design is still hard

426 points by the_mitsuhiko 11mo ago

callamdelaney 11mo ago
Agent design is still boring and I’m tired of hearing about it.
postalcoder 11mo ago
I've been building agent type stuff for a couple years now and the best thing I did was build my own framework and abstractions that I know like the back of my hand. I'd stay clear of any llm abstraction. There are so many companies with open source abstractions offering the panacea of a single interface that are crumbling under their own weight due to the sheer futility of supporting every permutation of every SDK evolution, all while the same companies try to build revenue generating businesses on top of them.
- the_mitsuhiko 11mo ago
  Author here. I’m with you on the abstractions part. I dumped a lot of my though so this into a follow up post: https://lucumr.pocoo.org/2025/11/22/llm-apis/ https://lucumr.pocoo.org/2025/11/22/llm-apis/
  - thierrydamiba 11mo ago
    Excellent write up. I’ve been thinking a lot about caching and agents so this was right ilup my alley. Have you experimented with using semantic cache on the chain of thought(what we get back from the providers anyways) and sending that to a dumb model for similar queries to “simulate” thinking?
  - msp26 11mo ago
    really nice post, will share!
- NitpickLawyer 11mo ago
  Yes, this is great advice. It also applies to interfaces. When we designed a support "chat bot", we went with a diferent architecture than what's out there already. We designed the system with "chat rooms" instead, and the frontend just dumps messages to a chatroom (with a session id). Then on the backend we can do lots of things, incrementally adding functionality, while the front end doesn't have to keep up. We can also do things like group messages, have "system" messages that other services can read, etc. It also feels more natural, as the client can type additional info while the server is working, etc. If you have to use some of the client side SDKs, another good idea is to have a proxy where you can also add functionality without having to change the frontend.
  - postalcoder 11mo ago
    Creativity is an underrated hard part of building agents. The fun part of building right now is knowing how little of the design space for building agents has been explored.
    - spacecadet 11mo ago
      This! I keep telling people that if tool use was not a an aha moment relative to AI agents for you, then you need to be more creative...
  - verdverm 11mo ago
    This is not so unlike the coding agent I'm building for vs code. One of the things I'm doing is keeping a snapshot of the current vs code state (files open, terminal history, etc) in the agent server. Similarly, I track the file changes without actually writing them until the user approves the diff, so there are some "filesystem" like things that need to be carefully managed on each side. tl;dr, Both sides are broadcasting messages and listening for the ones they care about.
- _pdp_ 11mo ago
  This is a huge undertaking though. Yes it is quite simple to build some basic abstraction on top of openai.complete or similar but this like 1% of an agent need to do. My bet is that agent frameworks and platform will become more like game engines. You can spin your own engine for sure and it is fun and rewarding. But AAA studios will most likely decide to use a ready to go platform with all the batteries included.
  - postalcoder 11mo ago
    In totality, yes. But you don't need every feature at once. You add to it once you hit boundaries. But I think the most important thing about this exercise is that you leave nothing to the imagination when building agents. The non-deterministic nature of LLMs already makes the performance of agents so difficult to interpret. Building agents on top of code that you cannot mentally trace through leads to so much frustration when addressing model underperformance and failure. It's hard to argue that after the dust settles, companies will default to batteries-included frameworks but, right now, a lot of people i've regretted adopting a large framework off the bat.
- sathish316 11mo ago
  I agree with your analysis of building your own Agent framework to have some level of control and fewer abstractions. Agents at their core are about programmatically talking to an LLM and performing these basic operations: 1. Structured Input and String Interpolation in prompts 2. Structured Output and Unmarshalling String response to Structured output (This is getting easier now with LLMs supporting Structured output) 3. Tool registry/discovery (of MCP and Function tools), Tool calls and response looping 4. Composability of Tools 5. Some form of Agent to Agent delegation I’ve had good luck with using PydanticAI which does these core operations well (due to the underlying Pydantic library), but still struggles with too many MCP servers/Tools and composability. I’ve built an open-source Agent framework called OpusAgents, that makes the process of creating Agents, Subagents, Tools that are simpler than MCP servers without overloading the context. Check it out here and tutorials/demos to see how it’s more reliable than generic Agents with MCP servers in Cursor/ClaudeDesktop - https://github.com/sathish316/opus_agents https://github.com/sathish316/opus_agents It’s built on top of PydanticAI and FastMCP, so that all non-core operations of Agents are accessible when I need them later.
  - spacecadet 11mo ago
    I also recommend this. I have tried all of the frameworks, and deploy some still for some clients- but for my personal agents, its my own custom framework that is dead simple and very easy to spin up, extend, etc.
  - drittich 11mo ago
    This sounds interesting. What about the agent behavior itself? How it decides how to come at a problem, what to show the user along the way, and how it decides when to stop? Are these things you have attempted to grapple with in your framework?
    - sathish316 11mo ago
      The framework has the following capabilities: 1. A way to create function tools 2. A way to create specialised subagents that can use their own tool or their own model. The main agent can delegate to subagent exposed as a tool. Subagents don’t get confused because they have their own context window, tools and even models (mix and match Remote LLM with Local LLM if needed) 3. Don’t use all tools of the MCP servers you’ve added. Filter out and select only the most relevant ones for the problem you’re trying to solve 4. HigherOrderTool is a way to callMCPTool(toolName, input) in places where the Agent to MCP interface can be better suited for the problem than what’s exposed as a generic interface by the MCP provider - https://github.com/sathish316/opus_agents/blob/main/docs/GUIDE_ADD_HIGHER_ORDER_TOOL.md https://github.com/sathish316/opus_agents/blob/main/docs/GUI... . This is similar to Anthropic’s recent blogpost on Code tools being better than MCP - https://www.anthropic.com/engineering/code-execution-with-mcp https://www.anthropic.com/engineering/code-execution-with-mc... 5. MetaTool is a way to use ready made patterns like OpenAPI and not having to write a tool or add more MCP servers to solve a problem - https://github.com/sathish316/opus_agents/blob/main/docs/GUIDE_ADD_META_TOOL.md https://github.com/sathish316/opus_agents/blob/main/docs/GUI... . This is similar to a recent HN post on Bash tools being better for context and accuracy than MCP - https://mariozechner.at/posts/2025-11-02-what-if-you-dont-need-mcp/ https://mariozechner.at/posts/2025-11-02-what-if-you-dont-ne... Other than AgentBuilder, CustomTool, HigherOrderTool, MetaTool, SubagentBuilder the framework does not try to control PydanticAI’s main agent behaviour. The high level approach is to use fewer, more relevant tools and let LLM orchestration and prompt tool references drive the rest. This approach has been more reliable and predictable for a given Agent based problem.
mritchie712 11mo ago
Some things we've[0] learned on agent design: 1. If your agent needs to write a lot of code, it's really hard to beat Claude Code (cc) / Agent SDK. We've tried many approaches and frameworks over the past 2 years (e.g. PydanticAI), but using cc is the first that has felt magic. 2. Vendor lock-in is a risk, but the bigger risk is having an agent that is less capable then what a user gets out of chatgpt because you're hand rolling every aspect of your agent. 3. cc is incredibly self aware. When you ask cc how to do something in cc, it instantly nails it. If you ask cc how to do something in framework xyz, it will take much more effort. 4. Give your agent a computer to use. We use e2b.dev, but Modal is great too. When the agent has a computer, it makes many complex features feel simple. 0 - For context, Definite (https://www.definite.app/ https://www.definite.app/) is a data platform with agents to operate it. It's like Heroku for data with a staff of AI data engineers and analysts.
- smcleod 11mo ago
  It's quite worrying that I have several times in the last few months had to really drive home why people should probably not be building bespoke agentic systems just to essentially act as a half baked version of an agentic coding tool when they could just go use Claude code and instead focus their efforts on creating value rather than instant technical debt.
  - CuriouslyC 11mo ago
    You can pretty much completely reprogram agents just by passing them through a smart proxy. You don't need to rewrite claude/codex, just add context engineering and tool behaviors at the proxy layer.
    - mritchie712 11mo ago
      yep, that's exactly where we've landed. focus on the tools and context, let claude handle the execution.
    - RamtinJ95 11mo ago
      This sounds very intriguing, any resources around this approach you can point me towards?
Fiveplus 11mo ago
I liked reading this but got a silly question as I am a noob in these things. If explicit caching is better, does that mean the agent is just forgetting stuff unless we manually save its notes? Are these things really that forgetful? Also why is there a virtual file system? So the agent is basically just running around a tiny digital desktop looking for its files? Why can't the agent just know where the data is? I'm sorry if these are juvenile questions.
- pjm331 11mo ago
  You are maybe confusing caching and context windows. Caching is mainly about keeping inference costs down
- the_mitsuhiko 11mo ago
  > If explicit caching is better, does that mean the agent is just forgetting stuff unless we manually save its notes? Caching is unrelated to memory, it's about how to not do the same work over and over again due to the distributed nature of state. I wrote a post that goes into detail from first principles here with my current thoughts on that topic [1]. > Are these things really that forgetful? No, not really. They can get side-tracked which is why most agents do a form of reinforcement in-context. > Why is there a virtual file system? So that you don't have dead-end tools. If a tool creates or manipulates state (which we represent on a virtual file system), another tool needs to be able to pick up the work. > Why can't the agent just know where the data is? And where would that data be? [1]: https://lucumr.pocoo.org/2025/11/22/llm-apis/ https://lucumr.pocoo.org/2025/11/22/llm-apis/
  - Fiveplus 11mo ago
    Thank you
CuriouslyC 11mo ago
The 'Reinforcement in the Agent Loop' section is a big deal, I use this pattern to enable async/event steered agents, it's super powerful. In long context you can use it to re-inject key points ("reminders"), etc.
- pjm331 11mo ago
  Yes that was the one point in here where I thought to myself oh yeah I’m going to go implement that immediately
srameshc 11mo ago
I still feel there is no sure shot way to build an abstraction yet. Probably that is why Loveable decided to build on Gemini AI rather than giving options of choosing model. On the other hand I like Pydantic AI framework and got myself a decent working solution where my preference is to stick with cheaper models by default and only use expensive only in cases where failure rate is too high.
- _pdp_ 11mo ago
  This is true. Based on my experience with real customers, they really don't know what is the difference between the different models. What they want is to get things done. The model is simply means to an end. As long as the task is completed, everything else is secondary.
  - morkalork 11mo ago
    I feel like there's always that "one guy" who has a preference and asks for an option to choose, and, it's a way for developers to offload the decision. You can't be blamed for the wrong one if you put that decision on the user. Tbh I agree with your approach and use it when building stuff for myself. I'm so over yak shaving that topic.
_pdp_ 11mo ago
I've started a company in this space about 2 years ago. We are doing fine. What we've learned so far is that a lot of these techniques are simply optimisations to tackle some deficiency in LLMs that is a problem "today". These are not going to be problems tomorrow because the technology will shift. As it happened many time in the span of the last 2 years. So yah, cool, caching all of that... but give it a couple of months and a better technique will come out - or more capable models. Many years ago when disc encryption on AWS was not an option, my team and I had to spend 3 months to come up with a way to encrypt the discs and do so well because at the time there was no standard way. It was very difficult as that required pushing encrypted images (as far as I remember). Soon after we started, AWS introduced standard disc encryption that you can turn on by clicking a button. We wasted 3 months for nothing. We should have waited! What I've learned from this is that often times it is better to do absolutely nothing.
- gchamonlive 11mo ago
  I think knowing when to do nothing is being able to evaluate if the problem the team is tackling is essential or tangential to the core focus of the project, and also whether the problem is something new or if it's been around for a while and there is still no standard way to solve it.
  - gessha 11mo ago
    Yeah, that will be the make it to brake it moment because if it’s too essential, it will be implemented but if it’s not, it may become a competitive advantage
- an0malous 11mo ago
  > These are not going to be problems tomorrow because the technology will shift. As it happened many time in the span of the last 2 years. What technology shifts have happened for LLMs in the last 2 years?
  - echelon 11mo ago
    We started putting them in image and video models and now image and video models are insane. I think the next period of high and rapid growth will be in media (image, video, sound, 3D), not text. It's much harder to adapt LLMs to solving business use cases with text. Each problem is niche, you have to custom tailor the solution, and the tooling is crude. The media use cases, by contrast, are low hanging fruit and result in 10,000x speedups and cost reductions almost immediately. The models are pure magic. I think more companies would be wise to ignore text for now and focus on visual domain problems. Nano Banana has so much more utility than agents. And there are so many low hanging fruit ways to make lots of money. Don't sleep on image and video. That's where the growth salient is.
Yokohiii 11mo ago
Most of it reads like too high expectation of overhyped technology.
- xpe 11mo ago
  What is the benefit of injecting one’s “take” on what level of hype exists when thinking about the OP’s exploration into agentic development? Better to step back and just look at what exists for what it is? Strive for a less biased take. The pieces are rapidly changing, non-standard, fragmented, and exploratory. If true, jumping in involves risks from chaos, churn, rushing, unproveness (sp?), etc
  - Yokohiii 11mo ago
    Confirmation bias can be a problem. Positivity bias as well. The author seems to take agentic dev as granted, which isn't. He doesn't goes into much detail either. It's at best a soft complaint about why it isn't plug and play already. I am open for the topic. But I demand more depth.
    - xpe 11mo ago
      Have you questioned confirmation bias on your own side as well?
      - Yokohiii 11mo ago
        About what? Most times i check, it's there. There is hype around AI and agentic dev. It's not subjective. My opinion is that it's a valid factor to consider if you evaluate tech. We've seen with i.e. microservices that people followed the hype because it seemed sound, but it was eventually terrible for many. Not everything is for everyone just because it's there. Can people explore things they care about? Yes. Can I have an opinion about what they do if they make a public post about it. Yes.
        xpe 11mo ago
cedws 11mo ago
Going to hijack this to ask a question that’s been on my mind: does anybody know why there’s no agentic tools that use tree-sitter to navigate code? It seems like it would be much more powerful than having the LLM grepping for strings or rewriting entire files to change one line.
- spacecadet 11mo ago
  Create an agent with these tools and you will. Agent tools are just functions, but think of them like skills. The more knowledge and skills your agents (plural, Id recommend more than one) have access to, the more useful they become.
- postalcoder 11mo ago
  This doesn't fully satisfy your question, but it comes straight from bcherny (claude code dev): > Claude Code doesn't use RAG currently. In our testing we found that agentic search out-performed RAG for the kinds of things people use Code for. source thread: https://news.ycombinator.com/item?id=43163011#43164253 https://news.ycombinator.com/item?id=43163011#43164253
  - cedws 11mo ago
    Thanks, I was also wondering why they don't use RAG.
    - postalcoder 11mo ago
      I forgot to mention that Aider uses a tree sitter approach. It's blazing fast and great for quick changes. It's not the greatest agent for doing heavy edits but I don't it has to do with them not using grep.
    - the_mitsuhiko 11mo ago
      They are using RAG. Grep is also just RAG. The better question is why they don’t use a vector database and honestly the answer is that these things are incredibly hard to keep in sync. And if they are not perfectly in sync, the utility drops quickly.
havkom 11mo ago
My tip is: don’t use SDK:s for agents. Use a while loop and craft your own JSON, handle context size and handle faults yourself. You will in practice need this level of control if you are not doing something trivial.
moinism 11mo ago
Amen. Been seeing these agent SDKs coming out left and right for a couple of years and thought it'd be a breeze to build an agent. Now I'm trying to build one for ~3 weeks, and I've tried three different SDKs and a couple of architectures. Here's what I found: - Claude Code SDK (now called Agent SDK) is amazing, but I think they are still in the process of decoupling it from the Claude Code, and that's why a few things are weird. e.g, You can define a subagent programmatically, but not skills. Skills have to be placed in the filesystem and then referenced in the plugin. Also, only Anthoripic models are supported :( - OpenAI's SDK's tight coupling with their platform is a plus point. i.e, you get agents and tool-use traces by default in your dashboard. Which you can later use for evaluation, distillation, or fine-tuning. But: 2. They have agent handoffs (which works in some cases), but not subagents. You can use tools as subagents, though. 1. Not easy to use a third-party model provider. Their docs provide sample codes, but it's not as easy as that. - Google Agent Kit doesn't provide any Typescript SDK yet. So didn't try. - Mastra, even though it looks pretty sweet, spins up a server for your agent, which you can then use via REST API. umm.. why? - SmythOS SDK is the one I'm currently testing because it provides flexibility in terms of choosing the model provider and defining your own architecture (handoffs or subagents, etc.). It has its quirks, but I think it'll work for now. Question: If you don't mind sharing, what is your current architecture? Agent -> SubAgents -> SubSubAgents? Linear? or a Planner-Executor? I'll write a detailed post about my learnings from architectures (fingers crossed) soon.
- mountainriver 11mo ago
  The frameworks are all pointless, just use AI assist to create agents in python or ideally a language with concurrency. You will be happy you did
  - moinism 11mo ago
    How do you deal with the different APIs/Tooluse schema in a custom build? As other people have mentioned, it's a bigger undertaking than it sounds.
    - catlover76 11mo ago
mitchellh 11mo ago
This is why I use the agent I use. I won't name the company, because I don't want people to think I'm a shill for them (I've already been accused of it before, but I'm just a happy, excited customer). But it's an agentic coding company that isn't associated with any of the big model providers. I don't want to keep up with all the new model releases. I don't want to read every model card. I don't want to feel pressured to update immediately (if it's better). I don't want to run evals. I don't want to think about when different models are better for different scenarios. I don't want to build obvious/common subagents. I don't want to manage N > 1 billing entities. I just want to work. Paying an agentic coding company to do this makes perfect sense for me.
- pjm331 11mo ago
  I’ve been surprised at the lack of discussion about sourcegraph’s Amp here which I’m pretty sure you’re referring to - it started a bit rough but these days I find that it’s really good
  - SatvikBeri 11mo ago
    So, I tried to sign up for Amp. I saw a livestream that mentioned you can sign up for their community Buildcrew on Discord and get $100 of credits. I tried signing up, and got an email that I was accepted and would soon get the credits. The Discord link did not work (it was expired) and the email was a noreply, so I tried emailing Amp support. This was last Friday (8 days ago.) As of today, no updated Discord link, no human response, no credits. If this is their norm, people probably aren't talking about it because they just haven't been able to try it.
    - sqs 11mo ago
      Sorry we missed that email! I don’t know what went wrong there, but I just replied and will figure it out. This is definitely not the norm (and Build Crew is a small fraction of our users).
      - SatvikBeri 11mo ago
lvl155 11mo ago
Design is hard because models change almost on a weekly basis. Quality abruptly falls off or drastically changes without announcements. It’s like building a house without proper foundation. I don’t know how many tokens I wasted because of this. I want to say 30% of the cost and resources this year for me.
jonmagic 11mo ago
Great post. Re: frameworks, I tried a number of them and then found Pocketflow and haven’t found a reason to try anything else since. It’s now been ported to half a dozen or more languages (including my port to Ruby). The simple api and mental model makes it easy for everyone on my team to jump into, extend, and compose. Highly recommend for anyone frustrated with the larger SDKs.
eclipsetheworld 11mo ago
We're repeating the same overengineering cycle we saw with early LangChain/RAG stacks. Just a couple of months ago the term agent was hard to define, but I've realized the best mental model is just a standard REPL: Read: Gather context (user input + tool outputs). Eval: LLM inference (decides: do I need a tool, or am I done?). Print: Execute the tool (the side effect) or return the answer. Loop: Feed the result back into the context window. Rolling a lightweight implementation around this concept has been significantly more robust for me than fighting with the abstractions in the heavy-weight SDKs.
- throw310822 11mo ago
  I don't think this has much to do with SDKs. I've developed my own agent code from scratch (starting from the simple loop) and eventually- unless your use case is really simple- you always have to deal with the need for subagents specialised for certain tasks, that share part of their data (but not all) with the main agent, with internal reasoning and reinforcement messages, etc.
  - eclipsetheworld 11mo ago
    Interestingly, sticking to the "Agent = REPL" mental model is actually what helped me solve those specific scaling problems (sub-agents and shared data) without the SDK bloat. 1. Sub-agents are just stack frames. When the main loop encounters a complex task, it "pushes" a new scope (a sub-agent with a fresh, empty context). That sub-agent runs its own REPL loop, returns only the clean result with out any context pollution and is then "popped". 2. Shared Data is the heap. Instead of stuffing "shared data" into the context window (which is expensive and confusing), I pass a shared state object by reference. Agents read/write to the heap via tools, but they only pass "pointers" in the conversation history. In the beginning this was just a Python dictionary and the "pointers" were keys. My issue with the heavy SDKs isn't that they try to solve these problems, but that they often abstract away the state management. I’ve found that explicitly managing the "stack" (context) and "heap" (artifacts) makes the system much easier to debug.
    - 11mo ago
rbren 11mo ago
If you want to hack on an emerging agent SDK (specializing in software development), join us at https://github.com/OpenHands/software-agent-sdk https://github.com/OpenHands/software-agent-sdk
ReDeiPirati 11mo ago
> We find testing and evals to be the hardest problem here. This is not entirely surprising, but the agentic nature makes it even harder. Unlike prompts, you cannot just do the evals in some external system because there’s too much you need to feed into it. This means you want to do evals based on observability data or instrumenting your actual test runs. So far none of the solutions we have tried have convinced us that they found the right approach here. I'm curious about the solutions the op has tried so far here.
- ColinEberhardt 11mo ago
  Likewise. I have a nasty feeling that most AI agent deployments happen with nothing more than some cursory manual testing. Going with the ‘vibes’ (to coin an over used term in the industry).
  - radarsat1 11mo ago
    A lot of "generative" work is like this. While you can come up with benchmarks galore, at the end of the day how a model "feels" only seems to come out from actual usage. Just read /r/localllama for opinions on which models are "benchmaxed" as they put it. It seems to be common knowledge in the local LLM community that many models perform well on benchmarks but that doesn't always reflect how good they actually are. In my case I was until recently working on TTS and this was a huge barrier for us. We used all the common signal quality and MOS-simulation models that judged so called "naturalness" and "expressiveness" etc. But we found that none of these really helped us much in deciding when one model was better than another, or when a model was "good enough" for release. Our internal evaluations correlated poorly with them, and we even disagreed quite a bit within the team on the quality of output. This made hyperparameter tuning as well as commercial planning extremely difficult and we suffered greatly for it. (Notice my use of past tense here..) Having good metrics is just really key and I'm now at the point where I'd go as far as to say that if good metrics don't exist it's almost not even worth working on something. (Almost.)
  - heljakka 11mo ago
    I can confirm this after hundreds of talks about the topic over the last 2 years. 90% of cases are simply not high-volume or high-stakes enough for the devs to care enough. I'm a founder of an evaluation automation startup, and our challenge is spotting teams right as their usage starts to grow and quality issues are about to escalate. Since that’s tough, we're trying to make the getting-to-first-evals so simple that teams can start building the mental models before things get out of hand.
deleted 11mo ago
[deleted]
ColinEberhardt 11mo ago
> We find testing and evals to be the hardest problem here … I wonder what this means for the agents that people are deploying into production? Are they tested at all? Or just manual ad-hoc testing? Sounds risky!
- verdverm 11mo ago
  I'm curious what people are doing. We're still very much in the experimentation phase > Sounds risky! One of first attempts at building file system tools for my custom agent called `tree` and caught a few node_models. Blew up my context and cost me $5 in 60s. Fortunately I triggered the TPM rate-limit and the thing stopped
nehalem 11mo ago
I am glad Vercel works on agents now. After all, Next is absolutely perfect and recommends them for greater challenges. /s
- benatkin 11mo ago
  From AWS wrapper to OpenAI wrapper
nowittyusername 11mo ago
If you are building an agent, start from scratch and build your own framework. This will save you more headache and wasted time down the line. one of the issues when you use someone else framework is that you miss out on learning and understanding important fundamentals about LLM's, how they work, context, etc... Also many developers don't learn the fundamentals of running LLM's locally and thus miss crucial context (giggidy) that would have helped them better understand the whole system. It seems to me that the author here came to a similar conclusion like many of us. I do want to add my own insights though that might be of use to some. One of the things he talked about was issues with reliable tool calling by the model. I recommend he try the following approach. Have the agent perform a self calibration exercise that makes the agent use his tools at the beginning of the context. Make him perform some complex stuff. Do it many times to test for coherence and accuracy while adjusting the system prompt towards more accurate tool calls. Once the agent had performed that calibration process successfully, you "freeze" that calibration context/history by broadening the --keep n to include not just the system prompt in the rolling window but also up to the end of this calibration session. then no matter how far the context window drifts the conditioned tokens generated by that calibration session steer the agent towards proper tool use. From then on your "system prompt" includes those turns. Note that this is probably not possible on cloud based models as you don't have access to the inference engine directly. A hacky way around that is emulate the conversation turns inside the system prompt. On the note of benchmark's. The calibration test is your benchmark from then on. When introducing new tools to the system prompt or adjusting any important variable, you must always rerun the same test to make sure the new adjustments you made don't negatively affect the system stability. On context engineering. That is a must as a bloated history will decohere the whole system. So its important to device an automated system that compresses the context down but retains overall essence of the history. there are about a billion ways you could do this and you will need to experiment a lot. LLM's are conditioned quite heavily from their own outputs, so having the ability to remove error tool calls from the context is a big boon as now the model is less likely to repeat its same mistakes. There are trade offs though, like he said caching is a no go when going this route but you gain a lot more control and stability within the whole system if you do this right. its basically reliability vs cost here. I tend to lean towards reliability. Also i don't recommend using the whole context size of the model. Most llms perform very poorly past a specific amount and I find that using maximum of 50% of the whole context window is recommended for cohesion. Meaning that if lets say max context window is 100k tokens, treat 50k as the max limit and start compressing the history around 35k tokens. Granular and step wise system can be set up. Where the most recent context is most detailed and uncompressed but as it goes further from the current time it gets less and less detailed. Obviously you want to store the full uncompressed history for a subagent that uses rag. This allows the agent to see in detail the whole thing if it finds the need to. ahh also on the matter of output. I found great success with making input and output channels for my agent. there are many channels that the model is conditioned in using for specific interactions. <think> channel for cot and reasoning. <message_to_user> channel for explicit messages to user. <call_agent> channel for calling agents and interacting with them. <call_tool> for tool use. and then a few other environment and system channels that are input channels from error scripts and environment towards the agent. This channel segmentation also allows for better management of internal automated scripts, and focus the model. Oh also one important thing is the fact that you need at least 2 separate output layers. meaning you need to separate your llm outputs from what is displayed to the user. and they have their own rules they follow. what that allows you to do is display information in a very human readable way to the real human while hiding all the noise but also retaining the crucial context thats needed for the model to function appropriately. bah anyways i rambled for long enough. good luck folks. hope this info helps someone.
Frannky 11mo ago
I use CLI Terminals as agent frameworks. You have big tech and open source behind them. All the problems are solved with zero work. They take care of new problems. You don't need to remove all the stuff that becomes outdated because the latest model doesn't make the same mistakes. Free via free tiers, cheap via openai compatible open source models like z.ai. Insanely smart and can easily talk to MCP servers.
munro 11mo ago
I had built an agent with LangGraph a 9 months ago--now seems React agents are in LangChain. Over all pretty happy with that, I just don't use any of the dumb stuff like embedding/search layer: just tools&state But I was actually playing with a few frameworks yesterday and struggling--I want what I want without having to write it. ;). Ended up using pydantic_ai package, literally just want tools w/ pydantic validation--but out of the box it doesn't have good observability, you would have to use their proprietary SaaS; and it comes bundled with Temporal.io (yo odio eso proyecto). I had to write my own observability which was annoying, and it sucks. If anyone has any things they've built, I would love to know, and TypeScript is an option. I want: - ReAct agent with tools that have schema validation - built in REALTIME observability w/ WebUI - customizable playground ChatUI (This is where TypeScript would shine) - no corporate takeover tentacles p.s.s: I know... I purposely try to avoid hard recommendations on HN, to avoid enshittification. "reddit best X" has been gamed. And generally skip these subtle promotional posts..
- thewhitetulip 11mo ago
  I have also worked on react agents using langgraph and I've not faced much issues! This thread has been confusing! Am I doing anything incorrectly? Or am I misunderstanding what people call as agents?
- DouweM 11mo ago
  Hey munro, Douwe from Pydantic AI here. Our docs have a page dedicated to observability: https://ai.pydantic.dev/logfire/ https://ai.pydantic.dev/logfire/, which is all based on OpenTelemetry and its gen_ai conventions. The Logfire SDK that Pydantic AI uses is a generic OTel client that defaults to sending data to Pydantic Logfire, our SaaS observability platform: https://pydantic.dev/logfire https://pydantic.dev/logfire, but can easily be pointed at any OTel sink: https://ai.pydantic.dev/logfire/#logfire-with-an-alternative-otel-backend https://ai.pydantic.dev/logfire/#logfire-with-an-alternative... Temporal is one of multiple durable execution solutions (https://ai.pydantic.dev/durable_execution/overview/ https://ai.pydantic.dev/durable_execution/overview/) we support and its SDK is indeed included by default in the "fat" `pydantic-ai` package, as are the SDKs for all model providers. There's also a `pydantic-ai-slim` package that doesn't include any optional dependencies: https://ai.pydantic.dev/install/#slim-install https://ai.pydantic.dev/install/#slim-install
elvin_d 11mo ago
My experience with BAML was really good. For structured outputs, JSON schema is slow and sluggish while BAML performs well. Interestingly don't see much attention on HN, wondering is it hype of other SDKs or BAML doesn't deliver enough.
dangoodmanUT 11mo ago
I think output functions aren't necessary, you should just use the text output when the agentic loop ends, and prompt to what kind of output you want (markdown, summary of changes, etc.)
jfghi 11mo ago
Technologically and mathematically this is all quite interesting. However, I have no desire to ever use an agent and I doubt there are many that will.
- kvirani 11mo ago
  No desire because you don't need them to optimize anything for yourself, or ?
  - jfghi 11mo ago
    So far the agentic stuff seems focused on shopping. Online shopping is remarkably seamless and optimized for me so the potential productivity gain is not enticing. I’ve encountered a number of errors dealing with LLMs so would be wary of the results. I also think there’d be an incentive to enshittify by having vendors pay to get preferential prioritization from the LLM. This could result in worse products being delivered for higher prices.
    - blharr 11mo ago
      Have you had issues with LLM agents or when chatting to them one-on-one?
    - xpe 11mo ago
      > So far the agentic stuff seems focused on shopping. Where are you getting this impression?
fudged71 11mo ago
If I understand correctly, Claude Code Agent SDK can’t edit itself (to incrementally improve), can’t edit or create its own agents and skills. I’ve found that by running Claude Code within Manus sandbox I can inspect the reasoning traces and edit the Agents/Skills with the Manus agent.
Vanclief 11mo ago
We have been working hard on the past two months implementing agents for different tasks. We started with Claude code, as we really liked working with hooks. However being vendor locked and having usage limit problems, we ended up implementing our own "runtime" that keeps the conversation structure while having hooks. Right now it only works with OpenAI but its designed to be able to incorporate Claude / Gemini down the road We ended up open sourcing that runtime if anyone is interested: https://github.com/Vanclief/agent-composer https://github.com/Vanclief/agent-composer
Scotrix 11mo ago
While working on a new startup I had exactly the same challenges and issues. As soon as it gets complex, data gets bigger, amount of tools increases it becomes tremendously difficult to control agents, network of agents or anything LLM related. Add then specific domains like legal or finance where most of the time there is just 0 or 1 and nothing in between (metaphorical speaking) it becomes a nightmare in code and side effects. So I started to actually build something to solve most of my own problems reliably and pushing deterministic outputs with help of LLMs (e.g. imagine finding the right columns/sheets in massive spreadsheets to create tool execution flows and fine tuning finding a range of data sources). My idea and solution which helped not only me but also quite a few business folks so far to fix and test agents is visualizing flows, connect and extract data visually, test and deploy changes in real time while keeping it very close to static types and predictable output (other than e.g. llama flow). Would love to hear your thoughts about my approach: https://llm-flow-designer.com https://llm-flow-designer.com
- wetpaws 11mo ago
  [dead]
cvhc 11mo ago
I'm frustrated that some fundamental aspects in agent development lack clear guideline. One example is input/output types of function tools. Frameworks offer some flexiblity and seemingly I can use fundamental types and simple data structures (list, dict/map). But on the other hand I know all data types are eventually stringified and this has implications. I have recently observed two issues when my agent calls a function that simply takes some int64 numeric IDs: (1) when the IDs are presented as hexadecimal in the context, the LLM attempts to convert them to decimal itself but mess it up because it doesn't really calculate; (2) some big IDs are not passed precisely in Google ADK framework [1], presumbly because its JSON serialization failed to keep the precision. I ended up changing the function to take string args instead. I also wasn't sure if the tool should return the data as original as possible in a moderately deeply nested dict, or step further to properly organize the output in a more human-readable text format for model ingestion. OpenAI's doc [2] writes: "A result must be a string, but the format is up to you (JSON, error codes, plain text, etc.). The model will interpret that string as needed." -- But that clearly contradicts with the framework's capability and some official examples where dict/numbers are returned. [1] https://github.com/google/adk-python/issues/3592 https://github.com/google/adk-python/issues/3592 [2] https://platform.openai.com/docs/guides/function-calling https://platform.openai.com/docs/guides/function-calling
linux2647 11mo ago
The tmux usage referenced at the end of the article was fascinating to watch. I’d never considered using tmux as a way of getting more insight into how an agent is working through a problem. Or to watch it debug something
cheriot 11mo ago
It's a library design flaw. The agent SDKs focus on an "easy" high level API and hard code all its assumptions (AI SDK, LangGraph, etc). There's no lower level primitives to recompose when you discover your requirements are different than what the library author thought of. So for now the choice is, "all in one for great prototypes and better hope it has everything you need" or roll your own. If someone knows of a library that's good for quick prototypes and malleable for power users please share.
bobwolf 11mo ago
Season 2 of the podcast "Shell Game" is about the host trying to do a startup company with only Ai agents and all the problems. It's interesting and entertaining
beeboop0 11mo ago
[dead]
wrochow 11mo ago
It seems to me that people are missing the point. AI is a tool, useful for some things, not so useful for others. The old adage applies here quite well. To a man with a hammer, everything is a nail. Having used AI quite extensively I have come to realize that to get quality outputs, it needs quality inputs... this is especially true with general models. It seems to me that many developers today just start typing without planning. That may work to some degree for humans, but AI needs more direction. In the early 2000s, Rational Unified Process (RUP) was all the rage. It gave way to Agile approaches, but now, I often wonder if we didn't throw the baby out with the bath water. I would wager that any AI model could produce high-quality code if provided with even a light version of RUP documentation.
d4rkp4ttern 11mo ago
Relatedly, I haven’t seen two issues discussed much in agent design, but repeatedly come up in real world use cases: (1) LLM forgets to call a tool (and instead outputs plain text). Contrary to some of the comments here saying that these concerns will disappear as frontier models improve, there will always be a need for having your agent scaffolding work well with weaker LLMs (cost, privacy, etc). (2) Determining when a task is finished. In some cases, we want the LLM to decide that (e.g search with different queries until desired info found), but in others, we want to specify deterministic task completion conditions (e.g., end the task immediately after structured info extraction, or after acting on such info, or after the LLM sees the result of that action etc). After repeatedly running into these types of issues in production agent systems, we’ve added mechanisms for these in the Langroid[1] agent framework (I’m the lead dev), which has blackboard-like loop architecture that makes it easy to incorporate these. For issue (1) we can configure an agent with a `handle_llm_no_tool` [2] set to a “nudge” that is sent back to the LLM when a non-tool response is detected (it could also be set as a lambda function to take other possible actions) For issue (2) Langroid has a DSL[3] for specifying task termination conditions. It lets you specify patterns that trigger task termination, e.g. - "T" to terminate immediately after a tool-call, - "T[X]" to terminate after calling the specific tool X, - "T,A" to terminate after a tool call, and agent handling (i.e. tool exec) - "T,A,L" to terminate after tool call, agent handling, and LLM response to that [1] Langroid https://github.com/langroid/langroid https://github.com/langroid/langroid [2] Handling non-tool LLM responses https://langroid.github.io/langroid/notes/handle-llm-no-tool/ https://langroid.github.io/langroid/notes/handle-llm-no-tool... [3] Task Termination in Langroid https://langroid.github.io/langroid/notes/task-termination/ https://langroid.github.io/langroid/notes/task-termination/
- d4rkp4ttern 11mo ago
  EDIT- forgot to mention the other part of issue (2): In cases where we want the LLM to decide task completion, in Langroid there's a DoneTool the LLM can use to signal completion. In general we find it useful to have orchestration tools for unambiguous control flow and message flow decisions by the LLM: https://langroid.github.io/langroid/reference/agent/tools/orchestration/ https://langroid.github.io/langroid/reference/agent/tools/or...
pjmlp 11mo ago
Of course it is "still", we are in baby steps in regards to technology adoption at industry scale.

426 points by the_mitsuhiko 11mo ago

Agent design is still boring and I’m tired of hearing about it.

I've been building agent type stuff for a couple years now and the best thing I did was build my own framework and abstractions that I know like the back of my hand. I'd stay clear of any llm abstraction. There are so many companies with open source abstractions offering the panacea of a single interface that are crumbling under their own weight due to the sheer futility of supporting every permutation of every SDK evolution, all while the same companies try to build revenue generating businesses on top of them.

the_mitsuhiko 11mo ago
Author here. I’m with you on the abstractions part. I dumped a lot of my though so this into a follow up post: https://lucumr.pocoo.org/2025/11/22/llm-apis/ https://lucumr.pocoo.org/2025/11/22/llm-apis/
- thierrydamiba 11mo ago
  Excellent write up. I’ve been thinking a lot about caching and agents so this was right ilup my alley. Have you experimented with using semantic cache on the chain of thought(what we get back from the providers anyways) and sending that to a dumb model for similar queries to “simulate” thinking?
- msp26 11mo ago
  really nice post, will share!
NitpickLawyer 11mo ago
Yes, this is great advice. It also applies to interfaces. When we designed a support "chat bot", we went with a diferent architecture than what's out there already. We designed the system with "chat rooms" instead, and the frontend just dumps messages to a chatroom (with a session id). Then on the backend we can do lots of things, incrementally adding functionality, while the front end doesn't have to keep up. We can also do things like group messages, have "system" messages that other services can read, etc. It also feels more natural, as the client can type additional info while the server is working, etc. If you have to use some of the client side SDKs, another good idea is to have a proxy where you can also add functionality without having to change the frontend.
- postalcoder 11mo ago
  Creativity is an underrated hard part of building agents. The fun part of building right now is knowing how little of the design space for building agents has been explored.
  - spacecadet 11mo ago
    This! I keep telling people that if tool use was not a an aha moment relative to AI agents for you, then you need to be more creative...
- verdverm 11mo ago
  This is not so unlike the coding agent I'm building for vs code. One of the things I'm doing is keeping a snapshot of the current vs code state (files open, terminal history, etc) in the agent server. Similarly, I track the file changes without actually writing them until the user approves the diff, so there are some "filesystem" like things that need to be carefully managed on each side. tl;dr, Both sides are broadcasting messages and listening for the ones they care about.
_pdp_ 11mo ago
This is a huge undertaking though. Yes it is quite simple to build some basic abstraction on top of openai.complete or similar but this like 1% of an agent need to do. My bet is that agent frameworks and platform will become more like game engines. You can spin your own engine for sure and it is fun and rewarding. But AAA studios will most likely decide to use a ready to go platform with all the batteries included.
- postalcoder 11mo ago
  In totality, yes. But you don't need every feature at once. You add to it once you hit boundaries. But I think the most important thing about this exercise is that you leave nothing to the imagination when building agents. The non-deterministic nature of LLMs already makes the performance of agents so difficult to interpret. Building agents on top of code that you cannot mentally trace through leads to so much frustration when addressing model underperformance and failure. It's hard to argue that after the dust settles, companies will default to batteries-included frameworks but, right now, a lot of people i've regretted adopting a large framework off the bat.
sathish316 11mo ago
I agree with your analysis of building your own Agent framework to have some level of control and fewer abstractions. Agents at their core are about programmatically talking to an LLM and performing these basic operations: 1. Structured Input and String Interpolation in prompts 2. Structured Output and Unmarshalling String response to Structured output (This is getting easier now with LLMs supporting Structured output) 3. Tool registry/discovery (of MCP and Function tools), Tool calls and response looping 4. Composability of Tools 5. Some form of Agent to Agent delegation I’ve had good luck with using PydanticAI which does these core operations well (due to the underlying Pydantic library), but still struggles with too many MCP servers/Tools and composability. I’ve built an open-source Agent framework called OpusAgents, that makes the process of creating Agents, Subagents, Tools that are simpler than MCP servers without overloading the context. Check it out here and tutorials/demos to see how it’s more reliable than generic Agents with MCP servers in Cursor/ClaudeDesktop - https://github.com/sathish316/opus_agents https://github.com/sathish316/opus_agents It’s built on top of PydanticAI and FastMCP, so that all non-core operations of Agents are accessible when I need them later.
- spacecadet 11mo ago
  I also recommend this. I have tried all of the frameworks, and deploy some still for some clients- but for my personal agents, its my own custom framework that is dead simple and very easy to spin up, extend, etc.
- drittich 11mo ago
  This sounds interesting. What about the agent behavior itself? How it decides how to come at a problem, what to show the user along the way, and how it decides when to stop? Are these things you have attempted to grapple with in your framework?
  - sathish316 11mo ago
    The framework has the following capabilities: 1. A way to create function tools 2. A way to create specialised subagents that can use their own tool or their own model. The main agent can delegate to subagent exposed as a tool. Subagents don’t get confused because they have their own context window, tools and even models (mix and match Remote LLM with Local LLM if needed) 3. Don’t use all tools of the MCP servers you’ve added. Filter out and select only the most relevant ones for the problem you’re trying to solve 4. HigherOrderTool is a way to callMCPTool(toolName, input) in places where the Agent to MCP interface can be better suited for the problem than what’s exposed as a generic interface by the MCP provider - https://github.com/sathish316/opus_agents/blob/main/docs/GUIDE_ADD_HIGHER_ORDER_TOOL.md https://github.com/sathish316/opus_agents/blob/main/docs/GUI... . This is similar to Anthropic’s recent blogpost on Code tools being better than MCP - https://www.anthropic.com/engineering/code-execution-with-mcp https://www.anthropic.com/engineering/code-execution-with-mc... 5. MetaTool is a way to use ready made patterns like OpenAPI and not having to write a tool or add more MCP servers to solve a problem - https://github.com/sathish316/opus_agents/blob/main/docs/GUIDE_ADD_META_TOOL.md https://github.com/sathish316/opus_agents/blob/main/docs/GUI... . This is similar to a recent HN post on Bash tools being better for context and accuracy than MCP - https://mariozechner.at/posts/2025-11-02-what-if-you-dont-need-mcp/ https://mariozechner.at/posts/2025-11-02-what-if-you-dont-ne... Other than AgentBuilder, CustomTool, HigherOrderTool, MetaTool, SubagentBuilder the framework does not try to control PydanticAI’s main agent behaviour. The high level approach is to use fewer, more relevant tools and let LLM orchestration and prompt tool references drive the rest. This approach has been more reliable and predictable for a given Agent based problem.

mritchie712 11mo ago

Some things we've[0] learned on agent design: 1. If your agent needs to write a lot of code, it's really hard to beat Claude Code (cc) / Agent SDK. We've tried many approaches and frameworks over the past 2 years (e.g. PydanticAI), but using cc is the first that has felt magic. 2. Vendor lock-in is a risk, but the bigger risk is having an agent that is less capable then what a user gets out of chatgpt because you're hand rolling every aspect of your agent. 3. cc is incredibly self aware. When you ask cc how to do something in cc, it instantly nails it. If you ask cc how to do something in framework xyz, it will take much more effort. 4. Give your agent a computer to use. We use e2b.dev, but Modal is great too. When the agent has a computer, it makes many complex features feel simple. 0 - For context, Definite (https://www.definite.app/ https://www.definite.app/) is a data platform with agents to operate it. It's like Heroku for data with a staff of AI data engineers and analysts.

smcleod 11mo ago
It's quite worrying that I have several times in the last few months had to really drive home why people should probably not be building bespoke agentic systems just to essentially act as a half baked version of an agentic coding tool when they could just go use Claude code and instead focus their efforts on creating value rather than instant technical debt.
- CuriouslyC 11mo ago
  You can pretty much completely reprogram agents just by passing them through a smart proxy. You don't need to rewrite claude/codex, just add context engineering and tool behaviors at the proxy layer.
  - mritchie712 11mo ago
    yep, that's exactly where we've landed. focus on the tools and context, let claude handle the execution.
  - RamtinJ95 11mo ago
    This sounds very intriguing, any resources around this approach you can point me towards?

Fiveplus 11mo ago

I liked reading this but got a silly question as I am a noob in these things. If explicit caching is better, does that mean the agent is just forgetting stuff unless we manually save its notes? Are these things really that forgetful? Also why is there a virtual file system? So the agent is basically just running around a tiny digital desktop looking for its files? Why can't the agent just know where the data is? I'm sorry if these are juvenile questions.

pjm331 11mo ago
You are maybe confusing caching and context windows. Caching is mainly about keeping inference costs down
the_mitsuhiko 11mo ago
> If explicit caching is better, does that mean the agent is just forgetting stuff unless we manually save its notes? Caching is unrelated to memory, it's about how to not do the same work over and over again due to the distributed nature of state. I wrote a post that goes into detail from first principles here with my current thoughts on that topic [1]. > Are these things really that forgetful? No, not really. They can get side-tracked which is why most agents do a form of reinforcement in-context. > Why is there a virtual file system? So that you don't have dead-end tools. If a tool creates or manipulates state (which we represent on a virtual file system), another tool needs to be able to pick up the work. > Why can't the agent just know where the data is? And where would that data be? [1]: https://lucumr.pocoo.org/2025/11/22/llm-apis/ https://lucumr.pocoo.org/2025/11/22/llm-apis/
- Fiveplus 11mo ago
  Thank you

CuriouslyC 11mo ago

The 'Reinforcement in the Agent Loop' section is a big deal, I use this pattern to enable async/event steered agents, it's super powerful. In long context you can use it to re-inject key points ("reminders"), etc.

pjm331 11mo ago
Yes that was the one point in here where I thought to myself oh yeah I’m going to go implement that immediately

srameshc 11mo ago

I still feel there is no sure shot way to build an abstraction yet. Probably that is why Loveable decided to build on Gemini AI rather than giving options of choosing model. On the other hand I like Pydantic AI framework and got myself a decent working solution where my preference is to stick with cheaper models by default and only use expensive only in cases where failure rate is too high.

_pdp_ 11mo ago
This is true. Based on my experience with real customers, they really don't know what is the difference between the different models. What they want is to get things done. The model is simply means to an end. As long as the task is completed, everything else is secondary.
- morkalork 11mo ago
  I feel like there's always that "one guy" who has a preference and asks for an option to choose, and, it's a way for developers to offload the decision. You can't be blamed for the wrong one if you put that decision on the user. Tbh I agree with your approach and use it when building stuff for myself. I'm so over yak shaving that topic.

_pdp_ 11mo ago

I've started a company in this space about 2 years ago. We are doing fine. What we've learned so far is that a lot of these techniques are simply optimisations to tackle some deficiency in LLMs that is a problem "today". These are not going to be problems tomorrow because the technology will shift. As it happened many time in the span of the last 2 years. So yah, cool, caching all of that... but give it a couple of months and a better technique will come out - or more capable models. Many years ago when disc encryption on AWS was not an option, my team and I had to spend 3 months to come up with a way to encrypt the discs and do so well because at the time there was no standard way. It was very difficult as that required pushing encrypted images (as far as I remember). Soon after we started, AWS introduced standard disc encryption that you can turn on by clicking a button. We wasted 3 months for nothing. We should have waited! What I've learned from this is that often times it is better to do absolutely nothing.

gchamonlive 11mo ago
I think knowing when to do nothing is being able to evaluate if the problem the team is tackling is essential or tangential to the core focus of the project, and also whether the problem is something new or if it's been around for a while and there is still no standard way to solve it.
- gessha 11mo ago
  Yeah, that will be the make it to brake it moment because if it’s too essential, it will be implemented but if it’s not, it may become a competitive advantage
an0malous 11mo ago
> These are not going to be problems tomorrow because the technology will shift. As it happened many time in the span of the last 2 years. What technology shifts have happened for LLMs in the last 2 years?
- echelon 11mo ago
  We started putting them in image and video models and now image and video models are insane. I think the next period of high and rapid growth will be in media (image, video, sound, 3D), not text. It's much harder to adapt LLMs to solving business use cases with text. Each problem is niche, you have to custom tailor the solution, and the tooling is crude. The media use cases, by contrast, are low hanging fruit and result in 10,000x speedups and cost reductions almost immediately. The models are pure magic. I think more companies would be wise to ignore text for now and focus on visual domain problems. Nano Banana has so much more utility than agents. And there are so many low hanging fruit ways to make lots of money. Don't sleep on image and video. That's where the growth salient is.

Yokohiii 11mo ago

Most of it reads like too high expectation of overhyped technology.

xpe 11mo ago
What is the benefit of injecting one’s “take” on what level of hype exists when thinking about the OP’s exploration into agentic development? Better to step back and just look at what exists for what it is? Strive for a less biased take. The pieces are rapidly changing, non-standard, fragmented, and exploratory. If true, jumping in involves risks from chaos, churn, rushing, unproveness (sp?), etc
- Yokohiii 11mo ago
  Confirmation bias can be a problem. Positivity bias as well. The author seems to take agentic dev as granted, which isn't. He doesn't goes into much detail either. It's at best a soft complaint about why it isn't plug and play already. I am open for the topic. But I demand more depth.
  - xpe 11mo ago
    Have you questioned confirmation bias on your own side as well?
    - Yokohiii 11mo ago
      About what? Most times i check, it's there. There is hype around AI and agentic dev. It's not subjective. My opinion is that it's a valid factor to consider if you evaluate tech. We've seen with i.e. microservices that people followed the hype because it seemed sound, but it was eventually terrible for many. Not everything is for everyone just because it's there. Can people explore things they care about? Yes. Can I have an opinion about what they do if they make a public post about it. Yes.
      - xpe 11mo ago

cedws 11mo ago

Going to hijack this to ask a question that’s been on my mind: does anybody know why there’s no agentic tools that use tree-sitter to navigate code? It seems like it would be much more powerful than having the LLM grepping for strings or rewriting entire files to change one line.

spacecadet 11mo ago
Create an agent with these tools and you will. Agent tools are just functions, but think of them like skills. The more knowledge and skills your agents (plural, Id recommend more than one) have access to, the more useful they become.
postalcoder 11mo ago
This doesn't fully satisfy your question, but it comes straight from bcherny (claude code dev): > Claude Code doesn't use RAG currently. In our testing we found that agentic search out-performed RAG for the kinds of things people use Code for. source thread: https://news.ycombinator.com/item?id=43163011#43164253 https://news.ycombinator.com/item?id=43163011#43164253
- cedws 11mo ago
  Thanks, I was also wondering why they don't use RAG.
  - postalcoder 11mo ago
    I forgot to mention that Aider uses a tree sitter approach. It's blazing fast and great for quick changes. It's not the greatest agent for doing heavy edits but I don't it has to do with them not using grep.
  - the_mitsuhiko 11mo ago
    They are using RAG. Grep is also just RAG. The better question is why they don’t use a vector database and honestly the answer is that these things are incredibly hard to keep in sync. And if they are not perfectly in sync, the utility drops quickly.

havkom 11mo ago

My tip is: don’t use SDK:s for agents. Use a while loop and craft your own JSON, handle context size and handle faults yourself. You will in practice need this level of control if you are not doing something trivial.

moinism 11mo ago

Amen. Been seeing these agent SDKs coming out left and right for a couple of years and thought it'd be a breeze to build an agent. Now I'm trying to build one for ~3 weeks, and I've tried three different SDKs and a couple of architectures. Here's what I found: - Claude Code SDK (now called Agent SDK) is amazing, but I think they are still in the process of decoupling it from the Claude Code, and that's why a few things are weird. e.g, You can define a subagent programmatically, but not skills. Skills have to be placed in the filesystem and then referenced in the plugin. Also, only Anthoripic models are supported :( - OpenAI's SDK's tight coupling with their platform is a plus point. i.e, you get agents and tool-use traces by default in your dashboard. Which you can later use for evaluation, distillation, or fine-tuning. But: 2. They have agent handoffs (which works in some cases), but not subagents. You can use tools as subagents, though. 1. Not easy to use a third-party model provider. Their docs provide sample codes, but it's not as easy as that. - Google Agent Kit doesn't provide any Typescript SDK yet. So didn't try. - Mastra, even though it looks pretty sweet, spins up a server for your agent, which you can then use via REST API. umm.. why? - SmythOS SDK is the one I'm currently testing because it provides flexibility in terms of choosing the model provider and defining your own architecture (handoffs or subagents, etc.). It has its quirks, but I think it'll work for now. Question: If you don't mind sharing, what is your current architecture? Agent -> SubAgents -> SubSubAgents? Linear? or a Planner-Executor? I'll write a detailed post about my learnings from architectures (fingers crossed) soon.

mountainriver 11mo ago
The frameworks are all pointless, just use AI assist to create agents in python or ideally a language with concurrency. You will be happy you did
- moinism 11mo ago
  How do you deal with the different APIs/Tooluse schema in a custom build? As other people have mentioned, it's a bigger undertaking than it sounds.
  - catlover76 11mo ago

mitchellh 11mo ago

This is why I use the agent I use. I won't name the company, because I don't want people to think I'm a shill for them (I've already been accused of it before, but I'm just a happy, excited customer). But it's an agentic coding company that isn't associated with any of the big model providers. I don't want to keep up with all the new model releases. I don't want to read every model card. I don't want to feel pressured to update immediately (if it's better). I don't want to run evals. I don't want to think about when different models are better for different scenarios. I don't want to build obvious/common subagents. I don't want to manage N > 1 billing entities. I just want to work. Paying an agentic coding company to do this makes perfect sense for me.

pjm331 11mo ago
I’ve been surprised at the lack of discussion about sourcegraph’s Amp here which I’m pretty sure you’re referring to - it started a bit rough but these days I find that it’s really good
- SatvikBeri 11mo ago
  So, I tried to sign up for Amp. I saw a livestream that mentioned you can sign up for their community Buildcrew on Discord and get $100 of credits. I tried signing up, and got an email that I was accepted and would soon get the credits. The Discord link did not work (it was expired) and the email was a noreply, so I tried emailing Amp support. This was last Friday (8 days ago.) As of today, no updated Discord link, no human response, no credits. If this is their norm, people probably aren't talking about it because they just haven't been able to try it.
  - sqs 11mo ago
    Sorry we missed that email! I don’t know what went wrong there, but I just replied and will figure it out. This is definitely not the norm (and Build Crew is a small fraction of our users).
    - SatvikBeri 11mo ago

lvl155 11mo ago

Design is hard because models change almost on a weekly basis. Quality abruptly falls off or drastically changes without announcements. It’s like building a house without proper foundation. I don’t know how many tokens I wasted because of this. I want to say 30% of the cost and resources this year for me.

jonmagic 11mo ago

Great post. Re: frameworks, I tried a number of them and then found Pocketflow and haven’t found a reason to try anything else since. It’s now been ported to half a dozen or more languages (including my port to Ruby). The simple api and mental model makes it easy for everyone on my team to jump into, extend, and compose. Highly recommend for anyone frustrated with the larger SDKs.

eclipsetheworld 11mo ago

We're repeating the same overengineering cycle we saw with early LangChain/RAG stacks. Just a couple of months ago the term agent was hard to define, but I've realized the best mental model is just a standard REPL: Read: Gather context (user input + tool outputs). Eval: LLM inference (decides: do I need a tool, or am I done?). Print: Execute the tool (the side effect) or return the answer. Loop: Feed the result back into the context window. Rolling a lightweight implementation around this concept has been significantly more robust for me than fighting with the abstractions in the heavy-weight SDKs.

throw310822 11mo ago
I don't think this has much to do with SDKs. I've developed my own agent code from scratch (starting from the simple loop) and eventually- unless your use case is really simple- you always have to deal with the need for subagents specialised for certain tasks, that share part of their data (but not all) with the main agent, with internal reasoning and reinforcement messages, etc.
- eclipsetheworld 11mo ago
  Interestingly, sticking to the "Agent = REPL" mental model is actually what helped me solve those specific scaling problems (sub-agents and shared data) without the SDK bloat. 1. Sub-agents are just stack frames. When the main loop encounters a complex task, it "pushes" a new scope (a sub-agent with a fresh, empty context). That sub-agent runs its own REPL loop, returns only the clean result with out any context pollution and is then "popped". 2. Shared Data is the heap. Instead of stuffing "shared data" into the context window (which is expensive and confusing), I pass a shared state object by reference. Agents read/write to the heap via tools, but they only pass "pointers" in the conversation history. In the beginning this was just a Python dictionary and the "pointers" were keys. My issue with the heavy SDKs isn't that they try to solve these problems, but that they often abstract away the state management. I’ve found that explicitly managing the "stack" (context) and "heap" (artifacts) makes the system much easier to debug.
  - 11mo ago

rbren 11mo ago

If you want to hack on an emerging agent SDK (specializing in software development), join us at https://github.com/OpenHands/software-agent-sdk https://github.com/OpenHands/software-agent-sdk

ReDeiPirati 11mo ago

> We find testing and evals to be the hardest problem here. This is not entirely surprising, but the agentic nature makes it even harder. Unlike prompts, you cannot just do the evals in some external system because there’s too much you need to feed into it. This means you want to do evals based on observability data or instrumenting your actual test runs. So far none of the solutions we have tried have convinced us that they found the right approach here. I'm curious about the solutions the op has tried so far here.

ColinEberhardt 11mo ago
Likewise. I have a nasty feeling that most AI agent deployments happen with nothing more than some cursory manual testing. Going with the ‘vibes’ (to coin an over used term in the industry).
- radarsat1 11mo ago
  A lot of "generative" work is like this. While you can come up with benchmarks galore, at the end of the day how a model "feels" only seems to come out from actual usage. Just read /r/localllama for opinions on which models are "benchmaxed" as they put it. It seems to be common knowledge in the local LLM community that many models perform well on benchmarks but that doesn't always reflect how good they actually are. In my case I was until recently working on TTS and this was a huge barrier for us. We used all the common signal quality and MOS-simulation models that judged so called "naturalness" and "expressiveness" etc. But we found that none of these really helped us much in deciding when one model was better than another, or when a model was "good enough" for release. Our internal evaluations correlated poorly with them, and we even disagreed quite a bit within the team on the quality of output. This made hyperparameter tuning as well as commercial planning extremely difficult and we suffered greatly for it. (Notice my use of past tense here..) Having good metrics is just really key and I'm now at the point where I'd go as far as to say that if good metrics don't exist it's almost not even worth working on something. (Almost.)
- heljakka 11mo ago
  I can confirm this after hundreds of talks about the topic over the last 2 years. 90% of cases are simply not high-volume or high-stakes enough for the devs to care enough. I'm a founder of an evaluation automation startup, and our challenge is spotting teams right as their usage starts to grow and quality issues are about to escalate. Since that’s tough, we're trying to make the getting-to-first-evals so simple that teams can start building the mental models before things get out of hand.

deleted 11mo ago

[deleted]

ColinEberhardt 11mo ago

> We find testing and evals to be the hardest problem here … I wonder what this means for the agents that people are deploying into production? Are they tested at all? Or just manual ad-hoc testing? Sounds risky!

verdverm 11mo ago
I'm curious what people are doing. We're still very much in the experimentation phase > Sounds risky! One of first attempts at building file system tools for my custom agent called `tree` and caught a few node_models. Blew up my context and cost me $5 in 60s. Fortunately I triggered the TPM rate-limit and the thing stopped

nehalem 11mo ago

I am glad Vercel works on agents now. After all, Next is absolutely perfect and recommends them for greater challenges. /s

benatkin 11mo ago
From AWS wrapper to OpenAI wrapper

nowittyusername 11mo ago

If you are building an agent, start from scratch and build your own framework. This will save you more headache and wasted time down the line. one of the issues when you use someone else framework is that you miss out on learning and understanding important fundamentals about LLM's, how they work, context, etc... Also many developers don't learn the fundamentals of running LLM's locally and thus miss crucial context (giggidy) that would have helped them better understand the whole system. It seems to me that the author here came to a similar conclusion like many of us. I do want to add my own insights though that might be of use to some. One of the things he talked about was issues with reliable tool calling by the model. I recommend he try the following approach. Have the agent perform a self calibration exercise that makes the agent use his tools at the beginning of the context. Make him perform some complex stuff. Do it many times to test for coherence and accuracy while adjusting the system prompt towards more accurate tool calls. Once the agent had performed that calibration process successfully, you "freeze" that calibration context/history by broadening the --keep n to include not just the system prompt in the rolling window but also up to the end of this calibration session. then no matter how far the context window drifts the conditioned tokens generated by that calibration session steer the agent towards proper tool use. From then on your "system prompt" includes those turns. Note that this is probably not possible on cloud based models as you don't have access to the inference engine directly. A hacky way around that is emulate the conversation turns inside the system prompt. On the note of benchmark's. The calibration test is your benchmark from then on. When introducing new tools to the system prompt or adjusting any important variable, you must always rerun the same test to make sure the new adjustments you made don't negatively affect the system stability. On context engineering. That is a must as a bloated history will decohere the whole system. So its important to device an automated system that compresses the context down but retains overall essence of the history. there are about a billion ways you could do this and you will need to experiment a lot. LLM's are conditioned quite heavily from their own outputs, so having the ability to remove error tool calls from the context is a big boon as now the model is less likely to repeat its same mistakes. There are trade offs though, like he said caching is a no go when going this route but you gain a lot more control and stability within the whole system if you do this right. its basically reliability vs cost here. I tend to lean towards reliability. Also i don't recommend using the whole context size of the model. Most llms perform very poorly past a specific amount and I find that using maximum of 50% of the whole context window is recommended for cohesion. Meaning that if lets say max context window is 100k tokens, treat 50k as the max limit and start compressing the history around 35k tokens. Granular and step wise system can be set up. Where the most recent context is most detailed and uncompressed but as it goes further from the current time it gets less and less detailed. Obviously you want to store the full uncompressed history for a subagent that uses rag. This allows the agent to see in detail the whole thing if it finds the need to. ahh also on the matter of output. I found great success with making input and output channels for my agent. there are many channels that the model is conditioned in using for specific interactions. <think> channel for cot and reasoning. <message_to_user> channel for explicit messages to user. <call_agent> channel for calling agents and interacting with them. <call_tool> for tool use. and then a few other environment and system channels that are input channels from error scripts and environment towards the agent. This channel segmentation also allows for better management of internal automated scripts, and focus the model. Oh also one important thing is the fact that you need at least 2 separate output layers. meaning you need to separate your llm outputs from what is displayed to the user. and they have their own rules they follow. what that allows you to do is display information in a very human readable way to the real human while hiding all the noise but also retaining the crucial context thats needed for the model to function appropriately. bah anyways i rambled for long enough. good luck folks. hope this info helps someone.

Frannky 11mo ago

I use CLI Terminals as agent frameworks. You have big tech and open source behind them. All the problems are solved with zero work. They take care of new problems. You don't need to remove all the stuff that becomes outdated because the latest model doesn't make the same mistakes. Free via free tiers, cheap via openai compatible open source models like z.ai. Insanely smart and can easily talk to MCP servers.

munro 11mo ago

I had built an agent with LangGraph a 9 months ago--now seems React agents are in LangChain. Over all pretty happy with that, I just don't use any of the dumb stuff like embedding/search layer: just tools&state But I was actually playing with a few frameworks yesterday and struggling--I want what I want without having to write it. ;). Ended up using pydantic_ai package, literally just want tools w/ pydantic validation--but out of the box it doesn't have good observability, you would have to use their proprietary SaaS; and it comes bundled with Temporal.io (yo odio eso proyecto). I had to write my own observability which was annoying, and it sucks. If anyone has any things they've built, I would love to know, and TypeScript is an option. I want: - ReAct agent with tools that have schema validation - built in REALTIME observability w/ WebUI - customizable playground ChatUI (This is where TypeScript would shine) - no corporate takeover tentacles p.s.s: I know... I purposely try to avoid hard recommendations on HN, to avoid enshittification. "reddit best X" has been gamed. And generally skip these subtle promotional posts..

thewhitetulip 11mo ago
I have also worked on react agents using langgraph and I've not faced much issues! This thread has been confusing! Am I doing anything incorrectly? Or am I misunderstanding what people call as agents?
DouweM 11mo ago
Hey munro, Douwe from Pydantic AI here. Our docs have a page dedicated to observability: https://ai.pydantic.dev/logfire/ https://ai.pydantic.dev/logfire/, which is all based on OpenTelemetry and its gen_ai conventions. The Logfire SDK that Pydantic AI uses is a generic OTel client that defaults to sending data to Pydantic Logfire, our SaaS observability platform: https://pydantic.dev/logfire https://pydantic.dev/logfire, but can easily be pointed at any OTel sink: https://ai.pydantic.dev/logfire/#logfire-with-an-alternative-otel-backend https://ai.pydantic.dev/logfire/#logfire-with-an-alternative... Temporal is one of multiple durable execution solutions (https://ai.pydantic.dev/durable_execution/overview/ https://ai.pydantic.dev/durable_execution/overview/) we support and its SDK is indeed included by default in the "fat" `pydantic-ai` package, as are the SDKs for all model providers. There's also a `pydantic-ai-slim` package that doesn't include any optional dependencies: https://ai.pydantic.dev/install/#slim-install https://ai.pydantic.dev/install/#slim-install

elvin_d 11mo ago

My experience with BAML was really good. For structured outputs, JSON schema is slow and sluggish while BAML performs well. Interestingly don't see much attention on HN, wondering is it hype of other SDKs or BAML doesn't deliver enough.

dangoodmanUT 11mo ago

I think output functions aren't necessary, you should just use the text output when the agentic loop ends, and prompt to what kind of output you want (markdown, summary of changes, etc.)

jfghi 11mo ago

Technologically and mathematically this is all quite interesting. However, I have no desire to ever use an agent and I doubt there are many that will.

kvirani 11mo ago
No desire because you don't need them to optimize anything for yourself, or ?
- jfghi 11mo ago
  So far the agentic stuff seems focused on shopping. Online shopping is remarkably seamless and optimized for me so the potential productivity gain is not enticing. I’ve encountered a number of errors dealing with LLMs so would be wary of the results. I also think there’d be an incentive to enshittify by having vendors pay to get preferential prioritization from the LLM. This could result in worse products being delivered for higher prices.
  - blharr 11mo ago
    Have you had issues with LLM agents or when chatting to them one-on-one?
  - xpe 11mo ago
    > So far the agentic stuff seems focused on shopping. Where are you getting this impression?

fudged71 11mo ago

If I understand correctly, Claude Code Agent SDK can’t edit itself (to incrementally improve), can’t edit or create its own agents and skills. I’ve found that by running Claude Code within Manus sandbox I can inspect the reasoning traces and edit the Agents/Skills with the Manus agent.

Vanclief 11mo ago

We have been working hard on the past two months implementing agents for different tasks. We started with Claude code, as we really liked working with hooks. However being vendor locked and having usage limit problems, we ended up implementing our own "runtime" that keeps the conversation structure while having hooks. Right now it only works with OpenAI but its designed to be able to incorporate Claude / Gemini down the road We ended up open sourcing that runtime if anyone is interested: https://github.com/Vanclief/agent-composer https://github.com/Vanclief/agent-composer

Scotrix 11mo ago

While working on a new startup I had exactly the same challenges and issues. As soon as it gets complex, data gets bigger, amount of tools increases it becomes tremendously difficult to control agents, network of agents or anything LLM related. Add then specific domains like legal or finance where most of the time there is just 0 or 1 and nothing in between (metaphorical speaking) it becomes a nightmare in code and side effects. So I started to actually build something to solve most of my own problems reliably and pushing deterministic outputs with help of LLMs (e.g. imagine finding the right columns/sheets in massive spreadsheets to create tool execution flows and fine tuning finding a range of data sources). My idea and solution which helped not only me but also quite a few business folks so far to fix and test agents is visualizing flows, connect and extract data visually, test and deploy changes in real time while keeping it very close to static types and predictable output (other than e.g. llama flow). Would love to hear your thoughts about my approach: https://llm-flow-designer.com https://llm-flow-designer.com

wetpaws 11mo ago
[dead]

cvhc 11mo ago

I'm frustrated that some fundamental aspects in agent development lack clear guideline. One example is input/output types of function tools. Frameworks offer some flexiblity and seemingly I can use fundamental types and simple data structures (list, dict/map). But on the other hand I know all data types are eventually stringified and this has implications. I have recently observed two issues when my agent calls a function that simply takes some int64 numeric IDs: (1) when the IDs are presented as hexadecimal in the context, the LLM attempts to convert them to decimal itself but mess it up because it doesn't really calculate; (2) some big IDs are not passed precisely in Google ADK framework [1], presumbly because its JSON serialization failed to keep the precision. I ended up changing the function to take string args instead. I also wasn't sure if the tool should return the data as original as possible in a moderately deeply nested dict, or step further to properly organize the output in a more human-readable text format for model ingestion. OpenAI's doc [2] writes: "A result must be a string, but the format is up to you (JSON, error codes, plain text, etc.). The model will interpret that string as needed." -- But that clearly contradicts with the framework's capability and some official examples where dict/numbers are returned. [1] https://github.com/google/adk-python/issues/3592 https://github.com/google/adk-python/issues/3592 [2] https://platform.openai.com/docs/guides/function-calling https://platform.openai.com/docs/guides/function-calling

linux2647 11mo ago

The tmux usage referenced at the end of the article was fascinating to watch. I’d never considered using tmux as a way of getting more insight into how an agent is working through a problem. Or to watch it debug something

cheriot 11mo ago

It's a library design flaw. The agent SDKs focus on an "easy" high level API and hard code all its assumptions (AI SDK, LangGraph, etc). There's no lower level primitives to recompose when you discover your requirements are different than what the library author thought of. So for now the choice is, "all in one for great prototypes and better hope it has everything you need" or roll your own. If someone knows of a library that's good for quick prototypes and malleable for power users please share.

bobwolf 11mo ago

Season 2 of the podcast "Shell Game" is about the host trying to do a startup company with only Ai agents and all the problems. It's interesting and entertaining

beeboop0 11mo ago

[dead]

wrochow 11mo ago

It seems to me that people are missing the point. AI is a tool, useful for some things, not so useful for others. The old adage applies here quite well. To a man with a hammer, everything is a nail. Having used AI quite extensively I have come to realize that to get quality outputs, it needs quality inputs... this is especially true with general models. It seems to me that many developers today just start typing without planning. That may work to some degree for humans, but AI needs more direction. In the early 2000s, Rational Unified Process (RUP) was all the rage. It gave way to Agile approaches, but now, I often wonder if we didn't throw the baby out with the bath water. I would wager that any AI model could produce high-quality code if provided with even a light version of RUP documentation.

d4rkp4ttern 11mo ago

Relatedly, I haven’t seen two issues discussed much in agent design, but repeatedly come up in real world use cases: (1) LLM forgets to call a tool (and instead outputs plain text). Contrary to some of the comments here saying that these concerns will disappear as frontier models improve, there will always be a need for having your agent scaffolding work well with weaker LLMs (cost, privacy, etc). (2) Determining when a task is finished. In some cases, we want the LLM to decide that (e.g search with different queries until desired info found), but in others, we want to specify deterministic task completion conditions (e.g., end the task immediately after structured info extraction, or after acting on such info, or after the LLM sees the result of that action etc). After repeatedly running into these types of issues in production agent systems, we’ve added mechanisms for these in the Langroid[1] agent framework (I’m the lead dev), which has blackboard-like loop architecture that makes it easy to incorporate these. For issue (1) we can configure an agent with a `handle_llm_no_tool` [2] set to a “nudge” that is sent back to the LLM when a non-tool response is detected (it could also be set as a lambda function to take other possible actions) For issue (2) Langroid has a DSL[3] for specifying task termination conditions. It lets you specify patterns that trigger task termination, e.g. - "T" to terminate immediately after a tool-call, - "T[X]" to terminate after calling the specific tool X, - "T,A" to terminate after a tool call, and agent handling (i.e. tool exec) - "T,A,L" to terminate after tool call, agent handling, and LLM response to that [1] Langroid https://github.com/langroid/langroid https://github.com/langroid/langroid [2] Handling non-tool LLM responses https://langroid.github.io/langroid/notes/handle-llm-no-tool/ https://langroid.github.io/langroid/notes/handle-llm-no-tool... [3] Task Termination in Langroid https://langroid.github.io/langroid/notes/task-termination/ https://langroid.github.io/langroid/notes/task-termination/

d4rkp4ttern 11mo ago
EDIT- forgot to mention the other part of issue (2): In cases where we want the LLM to decide task completion, in Langroid there's a DoneTool the LLM can use to signal completion. In general we find it useful to have orchestration tools for unambiguous control flow and message flow decisions by the LLM: https://langroid.github.io/langroid/reference/agent/tools/orchestration/ https://langroid.github.io/langroid/reference/agent/tools/or...

pjmlp 11mo ago

Of course it is "still", we are in baby steps in regards to technology adoption at industry scale.