4 ms·

> Is that just bad implementation? Where are the wasted tokens? How wouldn't it be wasteful? I'll try to summarize a couple sources: https://www.anthropic.co

by cstrahan 11mo ago

> Is that just bad implementation? Where are the wasted tokens? How wouldn't it be wasteful? I'll try to summarize a couple sources: https://www.anthropic.com/engineering/code-execution-with-mcp https://www.anthropic.com/engineering/code-execution-with-mc... https://blog.cloudflare.com/code-mode/ https://blog.cloudflare.com/code-mode/ Here's what Anthropic has to say about it: As MCP usage scales, there are two common patterns that can increase agent cost and latency: Tool definitions overload the context window; Intermediate tool results consume additional tokens. [...] Tool descriptions occupy more context window space, increasing response time and costs. In cases where agents are connected to thousands of tools, they’ll need to process hundreds of thousands of tokens before reading a request. [...] Most MCP clients allow models to directly call MCP tools. For example, you might ask your agent: "Download my meeting transcript from Google Drive and attach it to the Salesforce lead." The model will make calls like: TOOL CALL: gdrive.getDocument(documentId: "abc123") → returns "Discussed Q4 goals...\n[full transcript text]" (loaded into model context) TOOL CALL: salesforce.updateRecord( objectType: "SalesMeeting", recordId: "00Q5f000001abcXYZ", data: { "Notes": "Discussed Q4 goals...\n[full transcript text written out]" } ) (model needs to write entire transcript into context again) Every intermediate result must pass through the model. In this example, the full call transcript flows through twice. For a 2-hour sales meeting, that could mean processing an additional 50,000 tokens. Even larger documents may exceed context window limits, breaking the workflow. With large documents or complex data structures, models may be more likely to make mistakes when copying data between tool calls. Now, if you were to instead have the LLM write code, that code can perform whatever filtering/aggregation/transformation etc that it needs, without having to round-trip from LLM to tool(s), back and forth, and the only tokens that are consumed are those of the final result. What happens with MCP? All of the text of each MCP call is flooded into the context, only for the LLM to have to make sense of what it just read to then either regurgitate that out into a file to post process (very likely with differences/"hallucinations" slipped in), or in the usual case (I'm personifying the LLM here for rhetorical purposes) it simply tries to reason about what it read to give you the filtered/aggregated/transformed/etc result you're looking for -- again, very likely with mistakes made.

didibus 11mo ago
But none of the criticisms here is specific to MCP, just to tool calls in general, it wouldn't matter if the agent used a custom tool protocol, plain OpenAPIs, etc. These issues would still exist.
- cstrahan 11mo ago
  > These issues would still exist. I just explained why these issues don't apply to the LLM writing and invoking code: again, this is because code can apply successive transformations to the input without having to feed the intermediate results into the LLM's context. That code can read a file that would weigh in at 50,000 tokens, chain 100 functions together, producing a line that would be 20 tokens, and only the LLM will only see the final 20 token result. That really is only 20 tokens for the entire result -- the LLM never sees the 50,000 tokens from the file that was read via the program, nor does the LLM see the 10s of thousands of tokens worth of intermediate results between the successive transformations from each of the 100 functions. With MCP, there's no way for the LLM to invoke one tool call that expresses "compose/pipeline these 100 tools, please, and just give me the final result" -- the LLM must make 100 individual tool calls, manually threading the results through each tool call, which represents 100 opportunities for the LLM to make a mistake. It sounds like you are disagreeing with what I am saying, but it doesn't look you're giving any reason why you disagree, so I'm a bit confused.
  - didibus 11mo ago
    I'm not exactly disagreeing, its just that I don't think that's a criticism of MCP for tool calls. You need a way to expose tools to LLM, even for what you're talking about. The LLM can't write code without access to code writing tools. MCP is a protocol for exposing tools to an LLM agent in an extensible way. If you didn't have MCP for this, the alternative wouldn't be a python script. It would be some other tool calling protocol. Maybe a good criticism is that the protocol doesn't have a defined mechanism for piping tool calls together. It be nice to define as part of the protocol a standard way for the model to say call tool1 and then pass output.foo as input.bar of tool2. I feel that can probably be an eventual extension. As an advanced user of coding agents that you let run locally or have sandboxed yourself, you might get more power and mileage by having it implement scripts and execute them, or use bash/curl and so on, to say access the Google Drive standard APIs for whatever you want to do, than to use the Google Drive MCP. I 100% agree. But that's not adequate for all users, and it's not really an alternative to supporting custom tools in an agent.

didibus 11mo ago

But none of the criticisms here is specific to MCP, just to tool calls in general, it wouldn't matter if the agent used a custom tool protocol, plain OpenAPIs, etc. These issues would still exist.

cstrahan 11mo ago
> These issues would still exist. I just explained why these issues don't apply to the LLM writing and invoking code: again, this is because code can apply successive transformations to the input without having to feed the intermediate results into the LLM's context. That code can read a file that would weigh in at 50,000 tokens, chain 100 functions together, producing a line that would be 20 tokens, and only the LLM will only see the final 20 token result. That really is only 20 tokens for the entire result -- the LLM never sees the 50,000 tokens from the file that was read via the program, nor does the LLM see the 10s of thousands of tokens worth of intermediate results between the successive transformations from each of the 100 functions. With MCP, there's no way for the LLM to invoke one tool call that expresses "compose/pipeline these 100 tools, please, and just give me the final result" -- the LLM must make 100 individual tool calls, manually threading the results through each tool call, which represents 100 opportunities for the LLM to make a mistake. It sounds like you are disagreeing with what I am saying, but it doesn't look you're giving any reason why you disagree, so I'm a bit confused.
- didibus 11mo ago
  I'm not exactly disagreeing, its just that I don't think that's a criticism of MCP for tool calls. You need a way to expose tools to LLM, even for what you're talking about. The LLM can't write code without access to code writing tools. MCP is a protocol for exposing tools to an LLM agent in an extensible way. If you didn't have MCP for this, the alternative wouldn't be a python script. It would be some other tool calling protocol. Maybe a good criticism is that the protocol doesn't have a defined mechanism for piping tool calls together. It be nice to define as part of the protocol a standard way for the model to say call tool1 and then pass output.foo as input.bar of tool2. I feel that can probably be an eventual extension. As an advanced user of coding agents that you let run locally or have sandboxed yourself, you might get more power and mileage by having it implement scripts and execute them, or use bash/curl and so on, to say access the Google Drive standard APIs for whatever you want to do, than to use the Google Drive MCP. I 100% agree. But that's not adequate for all users, and it's not really an alternative to supporting custom tools in an agent.