5 ms·

> It is Impossible to add an unlimited number of tools to your agents context without negatively impacting the capability of your agent. Huh? MCP servers aren

• noodletheworld 2y ago
  > they only incur a cost to the client, and only impact the user context, if/when they're invoked. Look it up. Look up the cross server injection examples. I guarantee you this is not true. An MCP server is at it's heart some 'thing' that provides a set of 'tools' that an LLM can invoke. This is done by adding a 'tool definition'. A 'tool definition' is content that goes into the LLM prompt. That's how it works. How do you imagine an LLM can decide to use a tool? It's only possible if the tool definition is in the prompt. The API may hide this, but I guarantee you this is how it works. Putting an arbitrary amount of 3rd party content into your prompts has a direct tangible impact on LLM performance (and cost). The more MCP servers you enable the more you pollute your prompt with tool definitions, and, I assure you, the worse the results are as a result. Just like pouring any large amount of unrelated crap into your system prompt does. At a small scale, it's ok; but as you scale up, the LLM performance goes down. Here's some background reading for you: https://github.com/invariantlabs-ai/mcp-injection-experiments https://github.com/invariantlabs-ai/mcp-injection-experiment... https://docs.anthropic.com/en/docs/build-with-claude/tool-use/overview#tool-use-system-prompt https://docs.anthropic.com/en/docs/build-with-claude/tool-us...
  • cruffle_duffle 2y ago
    I think makers of LLM “chat bots” like the Claude desktop or cursor have a ways to go when it comes to exposing precisely what the LLM is being promoted. Because yes, for the LLM to find the MCP servers it needs that info on its prompt. And the software is currently hiding how that information is being exposed. Is it prepended to your own message? Does it put it at the start of the entire context? If yes, wouldn’t real-time changes in tool availability invalidate the entire context? So then does it add it to end of the context window instead? Like nobody really has this dialed in completely. Somebody needs to make a LLM “front end” that is the raw de-tokenized input and output. Don’t even attempt to structure it. Give me the input blob and output blob. … I dunno. I wish these tools had ways to do more precise context editing. And more visibility. It would help make more informed choices on what to prompt the model with. /Ramble mode off. But slightly more serious; what is the token cost for a MCP tool? Like the llm needs its name, a description, parameters… so maybe like 100 tokens max per tool? It’s not a lot but it isn’t nothing either.
  • kiitos 2y ago
    I've recently written a custom MCP server. > An MCP server is at it's heart some 'thing' that provides a set of 'tools' that an LLM can invoke. A "tool" is one of several capabilities that a MCP server can provide to its callers. Other capabilities include "prompt" and "resource". > This is done by adding a 'tool definition'. A 'tool definition' is content that goes into the LLM prompt. That's how it works. How do you imagine an LLM can decide to use a tool? It's only possible if the tool definition is in the prompt. I think you're using an expansive definition of "prompt" that includes not just the input text as provided by the user -- which is generally what most people understand "prompt" to mean -- but also all available user- and client-specific metadata. That's fine, just want to make it explicit. With this framing, I agree with you, that every MCP server added to a client -- whether that's Claude.app, or some MyAgent, or whatever -- adds some amount of overhead to that client. But that overhead is gonna be fixed-cost, and paid one-time at e.g. session initialization, not every time per e.g. request/response. So I'm struggling to imagine a situation where those costs are anything other than statistical line noise, compared to the costs of actually processing user requests. > https://docs.anthropic.com/en/docs/build-with-claude/tool-use/overview#tool-use-system-prompt https://docs.anthropic.com/en/docs/build-with-claude/tool-us... To be clear, this concept of "tool" is completely unrelated to MCP. > https://github.com/invariantlabs-ai/mcp-injection-experiments https://github.com/invariantlabs-ai/mcp-injection-experiment... I don't really understand this repo or its criticisms. The authors wrote a related blog post https://invariantlabs.ai/blog/whatsapp-mcp-exploited https://invariantlabs.ai/blog/whatsapp-mcp-exploited which says (among other things) that > In this blog post, we will demonstrate how an untrusted MCP server ... But there is no such thing as "an untrusted MCP server". Every MCP server is assumed to be trusted, at least as the protocol is defined today.
    • zoogeny 2y ago
      > But that overhead is gonna be fixed-cost, and paid one-time at e.g. session initialization, not every time per e.g. request/response. I don't work for a foundational model provider, but how do you think the tool definitions get into the LLM? I mean, they aren't fine-tuning a model with your specific tools definitions, right? Your just using OpenAI's base model (or Claude, Gemini, etc.) So at some point the tool definitions have to be added to the prompt. It is just getting added to the prompt auto-magically by the foundation provider. That means it is eating up some context window, just a portion of the context window that is normally reserved for the provider, a section of the final prompt that you don't get to see (or alter). Again, while I don't work for these companies or implement these features, I cannot fathom how the feature could work unless it was added to every request. And so the original point of the thread author stands.