5 ms·
I'm actually on the fence with skills. Vercel shared a study where they claimed skills performed actually worse [0] - than just injecting into the context direc
by neya 8mo ago
I'm actually on the fence with skills. Vercel shared a study where they claimed skills performed actually worse [0] - than just injecting into the context directly via agents.md. Similarly, there was a paper recently that suggested the same [1]
Of course, the classic response to these - even WITH the evidence is often "yOu'Re dOiNg iT wRonG". Does anyone actually have proof - where using skill.md is arguably better than not?
Edit: Fixed company name, added link to Vercel's claim
[0] https://vercel.com/blog/agents-md-outperforms-skills-in-our-agent-evals https://vercel.com/blog/agents-md-outperforms-skills-in-our-...
[1] https://arxiv.org/abs/2602.11988 https://arxiv.org/abs/2602.11988
- evalstate 8mo agoI think the paper is saying specifically that it's redundant to include information about your coding repository when that information is otherwise available to the agent in higher fidelity forms (e.g. package.json). This makes sense - but not sure it's about Skills directly. For the former I'd be interested in learning more about that. From a harness perspective the difference would be the inclusion of the description in the system prompt, and an additional tool call to return the skill. While that's certainly less efficient than adding the context directly I'd be surprised if it degraded task performance significantly. I tend to be quite focussed with my Skill/Tool usage in general though, inviting them in to context when needed rather than increasing the potential for model confusion.
- neya 8mo agoHere you go: Sorry, I miquoted the company, it was Vercel, not Cursor. "A compressed 8KB docs index embedded directly in AGENTS.md achieved a 100% pass rate, while skills maxed out at 79% even with explicit instructions telling the agent to use them. Without those instructions, skills performed no better than having no documentation at all." https://vercel.com/blog/agents-md-outperforms-skills-in-our-agent-evals https://vercel.com/blog/agents-md-outperforms-skills-in-our-...