3 ms·

I never really understood why you have to stuff all the tools in the context. Is there something wrong with having all your tools in, say, a markdown file, and

by losvedir 11mo ago

I never really understood why you have to stuff all the tools in the context. Is there something wrong with having all your tools in, say, a markdown file, and having a subagent read it with a description of the problem at hand and returning just the tool needed at that moment? Is that what this tool search is?

JyB 11mo ago
That’s exactly what it is in essence. The MCP protocol simply doesn’t have any mechanism specifications (yet) for not loading tools completely in the context. There’s nothing really strange about it. It’s just a protocol update issue.
- deleted 11mo ago
  [deleted]
falcor84 11mo ago
That's exactly what Claude Skills do [0], and while this tool search appears to be distinct, I do think that they're on the way to integrating MCP and Skills. [0] https://code.claude.com/docs/en/skills https://code.claude.com/docs/en/skills
- esperent 11mo ago
  I haven't had much luck with skills being called appropriately. When I have a skill called "X doer", and then I write a prompt like "Open <file> and do X", it almost never loads up the skill. I have to rewrite the prompt as "Open <file> and do X using the X doer skill". Which is basically exactly as much effort as what I was doing previously of having prewritten sub-prompts/agents in files and loading up the file each time I want to use it. I don't think this is an issue with how I'm writing skills, because it includes skill like the Skill Creator from Anthropic.
  - slhck 11mo ago
    Same experience here – it seems I have to specifically tell it to use the "X skill" to trigger it reliably. I guess with all the different rules set up for Claude to follow, it needs that particular word to draw its attention to the required skill.
    - _joel 11mo ago
      Ditto, I also find it'll invariably decide to disregard the CLAUDE.md again and produce a load of crap I didn't really ask it for.
  - notpublic 11mo ago
    Try adding a session hook that triggers on startup|resume|clear|compact to remind Claude about your custom skills.
    - esperent 11mo ago
      Is there a session start hook? I don't think so, unless it was added recently. I've mostly been working on smaller projects so I never need to compact. And skills are definitely not working even on the initial prompt of a new session.
jimbo808 11mo ago
Claude is pretty good at totally disregarding most of what’s in your CLAUDE.md, so I’m not optimistic. For example a project I work on gives it specific scripts to run when it runs automated tests, because the project is set up in a way that requires some special things to happen before tests will work correctly. I’ve never once seen it actually call those scripts on the first try. It always tries to run them using the typical command that doesn’t work with our setup, and I have to remind it the what correct thing to run is.
- snek_case 11mo ago
  I've had a similar experience with Gemini ignoring things I've explicitly told it (sometimes more than once). It's probably context rot. LLM give you a huge advertised number of tokens in the context, but the more stuff you put in there, the less reliably it remembers everything, which makes sense given how transformer attention blocks work internally.
- cerved 11mo ago
  Claude is pretty good at forgetting to run maven with -am flag, writing bash with heredocs that it's interpreter doesn't weird out on, using the != operator in jq. Maybe Claude has early onset dementia.
  - vendiddy 11mo ago
    Demented AIs running amock is just what we need in this day and age.
- notpublic 11mo ago
  Instead of including all these instructions in CLAUDE.md, have you considered using custom Skills? I’ve implemented something similar, and Skills works really well. The only downside is that it may consume more tokens.
  - taytus 11mo ago
    Yes, sometimes skills are more reliable, but not always. That is the biggest culprit to me so far. The fact that you cannot reliably trust these LLMs to follow steps or instructions makes them unsuitable for my applications.
noodletheworld 11mo ago
> I never really understood why you have to stuff all the tools in the context. You probably don't for... like, trivial cases? ...but, tool use is the most fine grained point, usually, in an agent's step-by-step implementation plan; So when planning, if you don't know what tool definitions exist, an agent might end up solving a problem naively step-by-step using primitive operations, when a single tool already exists that does that, or does part of it. Like, it's not quite as simple as "Hey, do X" It's more like: "Hey, make a plan to do X. When you're planning, first fetch a big list of the tools that seem vaguely related to the task and make a step-by-step plan keeping in mind the tools available to you" ...and then, for each step in the plan, you can do a tool search to find the best tool for x, then invoke it. Without a top level context of the tools, or tool categories, I think you'll end up in some dead-ends with agents trying to use very low level tools to do high level tasks and just spinning. The higher level your tool definitions are, the worse the problem is. I've found this is the case even now with MCP, where sometimes you have to explicitly tell an agent to use particular tools, not to try to re-invent stuff or use bash commands.

JyB 11mo ago

That’s exactly what it is in essence. The MCP protocol simply doesn’t have any mechanism specifications (yet) for not loading tools completely in the context. There’s nothing really strange about it. It’s just a protocol update issue.

deleted 11mo ago
[deleted]

falcor84 11mo ago

That's exactly what Claude Skills do [0], and while this tool search appears to be distinct, I do think that they're on the way to integrating MCP and Skills. [0] https://code.claude.com/docs/en/skills https://code.claude.com/docs/en/skills

esperent 11mo ago
I haven't had much luck with skills being called appropriately. When I have a skill called "X doer", and then I write a prompt like "Open <file> and do X", it almost never loads up the skill. I have to rewrite the prompt as "Open <file> and do X using the X doer skill". Which is basically exactly as much effort as what I was doing previously of having prewritten sub-prompts/agents in files and loading up the file each time I want to use it. I don't think this is an issue with how I'm writing skills, because it includes skill like the Skill Creator from Anthropic.
- slhck 11mo ago
  Same experience here – it seems I have to specifically tell it to use the "X skill" to trigger it reliably. I guess with all the different rules set up for Claude to follow, it needs that particular word to draw its attention to the required skill.
  - _joel 11mo ago
    Ditto, I also find it'll invariably decide to disregard the CLAUDE.md again and produce a load of crap I didn't really ask it for.
- notpublic 11mo ago
  Try adding a session hook that triggers on startup|resume|clear|compact to remind Claude about your custom skills.
  - esperent 11mo ago
    Is there a session start hook? I don't think so, unless it was added recently. I've mostly been working on smaller projects so I never need to compact. And skills are definitely not working even on the initial prompt of a new session.

jimbo808 11mo ago

Claude is pretty good at totally disregarding most of what’s in your CLAUDE.md, so I’m not optimistic. For example a project I work on gives it specific scripts to run when it runs automated tests, because the project is set up in a way that requires some special things to happen before tests will work correctly. I’ve never once seen it actually call those scripts on the first try. It always tries to run them using the typical command that doesn’t work with our setup, and I have to remind it the what correct thing to run is.

snek_case 11mo ago
I've had a similar experience with Gemini ignoring things I've explicitly told it (sometimes more than once). It's probably context rot. LLM give you a huge advertised number of tokens in the context, but the more stuff you put in there, the less reliably it remembers everything, which makes sense given how transformer attention blocks work internally.
cerved 11mo ago
Claude is pretty good at forgetting to run maven with -am flag, writing bash with heredocs that it's interpreter doesn't weird out on, using the != operator in jq. Maybe Claude has early onset dementia.
- vendiddy 11mo ago
  Demented AIs running amock is just what we need in this day and age.
notpublic 11mo ago
Instead of including all these instructions in CLAUDE.md, have you considered using custom Skills? I’ve implemented something similar, and Skills works really well. The only downside is that it may consume more tokens.
- taytus 11mo ago
  Yes, sometimes skills are more reliable, but not always. That is the biggest culprit to me so far. The fact that you cannot reliably trust these LLMs to follow steps or instructions makes them unsuitable for my applications.

noodletheworld 11mo ago

> I never really understood why you have to stuff all the tools in the context. You probably don't for... like, trivial cases? ...but, tool use is the most fine grained point, usually, in an agent's step-by-step implementation plan; So when planning, if you don't know what tool definitions exist, an agent might end up solving a problem naively step-by-step using primitive operations, when a single tool already exists that does that, or does part of it. Like, it's not quite as simple as "Hey, do X" It's more like: "Hey, make a plan to do X. When you're planning, first fetch a big list of the tools that seem vaguely related to the task and make a step-by-step plan keeping in mind the tools available to you" ...and then, for each step in the plan, you can do a tool search to find the best tool for x, then invoke it. Without a top level context of the tools, or tool categories, I think you'll end up in some dead-ends with agents trying to use very low level tools to do high level tasks and just spinning. The higher level your tool definitions are, the worse the problem is. I've found this is the case even now with MCP, where sometimes you have to explicitly tell an agent to use particular tools, not to try to re-invent stuff or use bash commands.