3 ms·
Adding an API call between my Agent and the LLM seems like it would kill latency. How much overhead does this add vs just managing the list locally?
by simonrogge 9mo ago
Adding an API call between my Agent and the LLM seems like it would kill latency. How much overhead does this add vs just managing the list locally?
- ofabioroma 9mo ago~20ms overhead. Built on Cloudflare's edge, so it's fast globally. The value isn't speed—it's that you stop rebuilding context infrastructure and get versioning/debugging for free.
- nicklo 9mo agoi've always wondered (for this, portkey, etc) - why not have a parallel option that fires an extra request instead of mitm the llm call?
- ofabioroma 9mo agoYou can fire them in parallel for simple cases. The issue is when you have multi-agent setups. If context isn't persisted before a sub-agent reads it, you get stale state. Single source of truth matters when agents are reading and writing to the same context. For single-agent flows, parallel works fine.