4 ms·
Show HN: Butter – A Behavior Cache for LLMs
Hi HN! I'm Erik. We built Butter, an LLM proxy that makes agent systems deterministic by caching and replaying responses, so automations behave consistently across runs.
- It’s a chat completions compatible endpoint, making it easy to drop into existing agents with a custom base_url
- The cache is template-aware, meaning lookups can treat dynamic content (names, addresses, etc.) as variables
You can see it in action in this demo where it memorizes tic-tac-toe games: https://www.youtube.com/watch?v=PWbyeZwPjuY https://www.youtube.com/watch?v=PWbyeZwPjuY
Why we built this: before Butter, we were Pig.dev (YC W25), where we built computer-use agents to automate legacy Windows applications. The goal was to replace RPA. But in practice, these agents were slow, expensive, and unpredictable - a major downgrade from deterministic RPA, and unacceptable in the worlds of healthcare, lending, and government. We realized users don't want to replace RPA with AI, they just want AI to handle the edge cases.
We set out to build a system for "muscle memory" for AI automations (general purpose, not just computer-use), where agent trajectories get baked into reusable code. You may recall our first iteration of this in May, a library called Muscle Mem: https://news.ycombinator.com/item?id=43988381 https://news.ycombinator.com/item?id=43988381
Today we're relaunching it as a chat completions proxy. It emulates scripted automations by storing observed message histories in a tree structure, where each fork in the tree represents some conditional branch in the workflow's "code". We replay behaviors by walking the agent down the tree, falling back to AI to add new branches if the next step is not yet known.
The proxy is live and free to use while we work through making the template-aware engine more flexible and accurate. Please try it out and share how it went, where it breaks, and if it’s helpful.
- robofanatic 11mo agoSo instead of OpenAI I should pay butter?
- edunteman 11mo agoIt’s bring-your-own-key, so any calls proxied to OpenAI just end up billing directly to your account as normal. You’d only pay Butter for calls that don’t go to the provider. That’d be a separate billing account with butter.
- realitysballs 11mo agoFunny, we are working to implement this same logic in our in-house financial categorization agent. When we have a repeat prompt it goes to a json that stores answers and only goes to AI for edge cases. It’s a good idea
- edunteman 11mo agoAwesome to hear you’ve done similar. JSON artifacts from runs seem to be a common approach for building this in house, similar to what we did with the muscle mem. Detecting cache misses is a bit hard without seeing what the model sees, part of what inspired this proxy direction. Thanks for the nice words!
- ronbenton 11mo agoInteresting... is it legal?
- edunteman 11mo agoI couldn’t see how it wouldn’t be, as it’s a free market opt-in decision to use Butter
- ronbenton 11mo agoit wouldn't be the first API service to disallow someone from selling a cache layer for their API. After all, this should likely result in OpenAI (or whatever provider) making less money
- edunteman 11mo agoAh yes that makes sense, have heard of those cases too but hadn’t put much thought into it. Thanks for pointing it out!
- RestartKernel 11mo agoI've seen the OpenRouter guys here on HN before, so you can probably ask them what to look out for.
- puppycodes 11mo agoI like the pricing model but I'm skeptical it will last.
- edunteman 11mo agoI feel the same - we’ll use it as long as we can since it’s customer aligned but I wouldn’t be surprised if competitive or COGs costs force us to change in the future.
- Jayakumark 11mo agoWhat local models will it work with ? Also what will be the pricing for local llms?
- edunteman 11mo agoGood question, I imagine you’d need to set up an ngrok endpoint to tunnel to local LLMs. In those cases perhaps an open source (maybe even local) version would make more sense. For our hosted version we’d need to charge something, given storage requirements to run such a service, but especially for local models that feels wrong. I’ve been considering open source for this reason.
- invisibleink 11mo agointeresting. is the answer not context specific most of the time? even if I ask LLM the same question again and again the answer depends on the context. what are some use cases where you need deterministic caching?
- barapa 11mo agoWe often will repeat calls to try again. Or sometimes we make the same call multiple times to get multiple answers and then score or merge them. Is this used only in cases where you assume the answer from your first call is correct?
- edunteman 11mo agoI’d love your opinion here! Right now, we assume first call is correct, and will eagerly take the first match we find while traversing the tree. One of the worst things that could currently happen is we cache a bad run, and now instead of occasional failures you’re given 100% failures. A few approaches we’ve considered - maintain a staging tree, and only promote to live if multiple sibling nodes (messages) look similar enough. Decision to promote could be via tempting, regex, fuzzy, semantic, or LLM-judged - add some feedback APIs for a client to score end-to-end runs so that path could develop some reputation
- toobulkeh 11mo agoI’d assume RL would be baked in to the request structure. I’m surprised OAI spec doesn’t include it, but I suppose you could hijack a conversation flow to do so
- mountainriver 11mo agoI also did computer agents with a vc backed startup, ran into the same issues, and we built a fairly similar thing at one point. It’s useful but it has limitations, it seems to only work well in environments that are perfectly predictable otherwise it gets in the way of the agent. I think I prefer RL over these approaches but it requires a bit more data.
- rajit 11mo agoWe spoke to a number of browser agent companies who said deterministic RPA with an AI fallback was their "secret" :)
- edunteman 11mo agoVery, very common approach! Wrote more on that here: https://blog.butter.dev/the-messy-world-of-deterministic-agents https://blog.butter.dev/the-messy-world-of-deterministic-age...
- toobulkeh 11mo agoWhat a great overview! I’d love your thoughts on my addition, autolearn.dev — voyager behind MCP. The proxy format is exactly what I needed! Thanks
- banjwoorri 11mo ago[dead]
- sherinjosephroy 11mo ago[flagged]
- felipe-pathwave 11mo agoany plans to support https://openrouter.ai/ https://openrouter.ai/ ?