5 ms·

Folks have run comparisons. From a huggingface employee: codex + skills finetunes Qwen3-0.6B to +6 on humaneval and beats the base score on the first run.

Folks have run comparisons. From a huggingface employee: codex + skills finetunes Qwen3-0.6B to +6 on humaneval and beats the base score on the first run. I reran the experiment from this week, but used codex's new skills integration. Like claude code, codex consumes the full skill into context and doesn't start with failing runs. It's first run beats the base score, and on the second run it beats claude code. https://xcancel.com/ben_burtenshaw/status/2000233069517676756 https://xcancel.com/ben_burtenshaw/status/200023306951767675... That said, it's not a perfect comparison because of the Codex model mismatch between runs. The author seems to be doing a lot of work on skills evaluation. https://github.com/huggingface/upskill https://github.com/huggingface/upskill

xrd 8mo ago
Does this indicate running locally with a very small (quantized?) model? I am very interested in finding ways to combine skills + local models + MCP + aider-ish tools to avoid using commercial LLM providers. Is this a path to follow? Or, something different?
- postalcoder 8mo ago
  Check out the guy's work. He's doing a lot of work on precisely what you're talking about. https://xcancel.com/ben_burtenshaw https://xcancel.com/ben_burtenshaw https://huggingface.co/blog/upskill https://huggingface.co/blog/upskill https://github.com/huggingface/upskill https://github.com/huggingface/upskill
pton_xd 8mo ago
I think the point is it smells like a hack, just like "think extra hard and I'll tip you $200" was a few years ago. It increases benchmarks a few points now but what's the point in standardizing all this if it'll be obsolete next year?
- deleted 8mo ago
  [deleted]
- mbesto 8mo ago
  I think this tweet sums it correctly doesn't? A +6 jump on a 0.6B model is actually more impressive than a +2 jump on a 100B model. It proves that 'intelligence' isn't just parameter count; it is context relevance. You are proving that a lightweight model with a cheat sheet beats a giant with amnesia. This is the death of the 'bigger is better' dogma Which is essentially the bitter lesson that Richard Sutton talks about?
  - Der_Einzige 8mo ago
    Nice ChatGPT generated response in that tweet. Anyone too lazy to deslop their tweet shouldn't be listened to.
- 9dev 8mo ago
  Standards have to start somewhere to gain traction and proliferate themselves for longer than that. Plus, as has been mentioned multiple times here, standard skills are a lot more about different harnesses being able to consistently load skills into the context window in a programmatic way. Not every AI workload is a local coding agent.
- dragonwriter 8mo ago
  The standardization is for presentation of how the information is made available to the harness. Optimizations in how the information is presented to the model can be iterated on without impacting the presentation to the harness. Initially, agent skills have already been provided by: (1) providing a bash tool with direct access to the filesystem storing the skills to the model, (2) providing read_file and related tools to the model, (3) by providing specialized tools to access skills to the model, (4) by processing the filesystem structure and providing a structure that includes the full content of the skills up front to the model. And probably some other ways or hybrids. > It increases benchmarks a few points now but what's the point in standardizing all this if it'll be obsolete next year? Standardizing the information presentation of skills to LLM harnesses lets the harnesses incorporate findings on optimization (which may be specific to models, or at least model features like context size, and use cases) and existing skills getting the benefit of that for free.
  - 0thgen 8mo ago
    How much of a standard is it though, really? To me it just looks like "Call your docs SKILLS and organize it like this". And if you're just making docs and letting your models go buck wild in your shell, doesn't an overspecified docs structure ruin the point of general purpose agents? Like, a good dev should be able to walk into a codebase, look at the structure, and figure out how to proceed. If "hey your docs aren't where I was expecting" breaks the developer, you shouldn't have hired them. Feels like a weird thing to take "this is how we organize our repos as this company" and turn that into "this is an 'open standard' that you should build your workflows around".
8cvor6j844qw_d6 8mo ago
Sounds like the benchmark matrix just got a lot bigger, model * skill combinations.
iainmerrick 8mo ago
I can't quite tell what's being compared there -- just looks like several different LLMs? To be clear, I'm suggesting that any specific format for "skills.md" is a red herring, and all you need to do is provide the LLM with good clear documentation. A useful comparison would be between: a) make a carefully organised .skills/ folder, b) put the same info anywhere and just link to it from your top-level doc, c) just dump everything directly in the top-level doc. My guess is that it's probably a good idea to break stuff out into separate sections, to avoid polluting the context with stuff you don't need; but the specific way you do that very likely isn't important at all. So (a) and (b) would perform about the same.
- postalcoder 8mo ago
  Your skepticism is valid. Vercel ran a study where they said that skills underperform putting a docs index in AGENTS.md[0]. My guess is that the standardization is going to make its way into how the models are trained and Skills are eventually going to pull out ahead. 0: https://vercel.com/blog/agents-md-outperforms-skills-in-our-agent-evals https://vercel.com/blog/agents-md-outperforms-skills-in-our-...
  - vidarh 8mo ago
    Agents add a docs index in context for skills, so this is an issue of finding that the current specific implementation of skills in Claude Code is suboptimal. Their reasoning about it is also flawed. E.g. "No decision point. With AGENTS.md, there's no moment where the agent must decide "should I look this up?" The information is already present." - but this is exactly the case for skills too. The difference is just where in the context the information is, and how it is structured. Having looked at their article, ironically I think the reason it works is that they likely force more information into context by giving the agent less information to work with: Instead of having a description, which might convince the agent a given skill isn't relevant, their index is basically a list of vague filenames, forcing the agent to make a guess, and potentialy reading the wrong thing. This is basically exactly what skills were added to avoid. But it will break if the description isn't precise enough. And it's perfectly possible that current tooling isn't aggressive enough about pruning detail that might tempt the agent to ignore relevant files.
bburtenshaw 8mo ago
thanks for sharing the work. correct, we're currently working on evals for skills so you can compare skills between models and harnesses. we wrote a blog on getting agents to write CUDA kernels and evaluating them: https://huggingface.co/blog/upskill https://huggingface.co/blog/upskill
oofbey 8mo ago
This is a neat idea for a test. But the test is badly executed. A single comparison could just be a fluke. Compare it on a dozen tasks, trying each task a dozen times. Then you get data which is believable.

xrd 8mo ago

Does this indicate running locally with a very small (quantized?) model? I am very interested in finding ways to combine skills + local models + MCP + aider-ish tools to avoid using commercial LLM providers. Is this a path to follow? Or, something different?

postalcoder 8mo ago
Check out the guy's work. He's doing a lot of work on precisely what you're talking about. https://xcancel.com/ben_burtenshaw https://xcancel.com/ben_burtenshaw https://huggingface.co/blog/upskill https://huggingface.co/blog/upskill https://github.com/huggingface/upskill https://github.com/huggingface/upskill

pton_xd 8mo ago

I think the point is it smells like a hack, just like "think extra hard and I'll tip you $200" was a few years ago. It increases benchmarks a few points now but what's the point in standardizing all this if it'll be obsolete next year?

deleted 8mo ago
[deleted]
mbesto 8mo ago
I think this tweet sums it correctly doesn't? A +6 jump on a 0.6B model is actually more impressive than a +2 jump on a 100B model. It proves that 'intelligence' isn't just parameter count; it is context relevance. You are proving that a lightweight model with a cheat sheet beats a giant with amnesia. This is the death of the 'bigger is better' dogma Which is essentially the bitter lesson that Richard Sutton talks about?
- Der_Einzige 8mo ago
  Nice ChatGPT generated response in that tweet. Anyone too lazy to deslop their tweet shouldn't be listened to.
9dev 8mo ago
Standards have to start somewhere to gain traction and proliferate themselves for longer than that. Plus, as has been mentioned multiple times here, standard skills are a lot more about different harnesses being able to consistently load skills into the context window in a programmatic way. Not every AI workload is a local coding agent.
dragonwriter 8mo ago
The standardization is for presentation of how the information is made available to the harness. Optimizations in how the information is presented to the model can be iterated on without impacting the presentation to the harness. Initially, agent skills have already been provided by: (1) providing a bash tool with direct access to the filesystem storing the skills to the model, (2) providing read_file and related tools to the model, (3) by providing specialized tools to access skills to the model, (4) by processing the filesystem structure and providing a structure that includes the full content of the skills up front to the model. And probably some other ways or hybrids. > It increases benchmarks a few points now but what's the point in standardizing all this if it'll be obsolete next year? Standardizing the information presentation of skills to LLM harnesses lets the harnesses incorporate findings on optimization (which may be specific to models, or at least model features like context size, and use cases) and existing skills getting the benefit of that for free.
- 0thgen 8mo ago
  How much of a standard is it though, really? To me it just looks like "Call your docs SKILLS and organize it like this". And if you're just making docs and letting your models go buck wild in your shell, doesn't an overspecified docs structure ruin the point of general purpose agents? Like, a good dev should be able to walk into a codebase, look at the structure, and figure out how to proceed. If "hey your docs aren't where I was expecting" breaks the developer, you shouldn't have hired them. Feels like a weird thing to take "this is how we organize our repos as this company" and turn that into "this is an 'open standard' that you should build your workflows around".

8cvor6j844qw_d6 8mo ago

Sounds like the benchmark matrix just got a lot bigger, model * skill combinations.

iainmerrick 8mo ago

I can't quite tell what's being compared there -- just looks like several different LLMs? To be clear, I'm suggesting that any specific format for "skills.md" is a red herring, and all you need to do is provide the LLM with good clear documentation. A useful comparison would be between: a) make a carefully organised .skills/ folder, b) put the same info anywhere and just link to it from your top-level doc, c) just dump everything directly in the top-level doc. My guess is that it's probably a good idea to break stuff out into separate sections, to avoid polluting the context with stuff you don't need; but the specific way you do that very likely isn't important at all. So (a) and (b) would perform about the same.

postalcoder 8mo ago
Your skepticism is valid. Vercel ran a study where they said that skills underperform putting a docs index in AGENTS.md[0]. My guess is that the standardization is going to make its way into how the models are trained and Skills are eventually going to pull out ahead. 0: https://vercel.com/blog/agents-md-outperforms-skills-in-our-agent-evals https://vercel.com/blog/agents-md-outperforms-skills-in-our-...
- vidarh 8mo ago
  Agents add a docs index in context for skills, so this is an issue of finding that the current specific implementation of skills in Claude Code is suboptimal. Their reasoning about it is also flawed. E.g. "No decision point. With AGENTS.md, there's no moment where the agent must decide "should I look this up?" The information is already present." - but this is exactly the case for skills too. The difference is just where in the context the information is, and how it is structured. Having looked at their article, ironically I think the reason it works is that they likely force more information into context by giving the agent less information to work with: Instead of having a description, which might convince the agent a given skill isn't relevant, their index is basically a list of vague filenames, forcing the agent to make a guess, and potentialy reading the wrong thing. This is basically exactly what skills were added to avoid. But it will break if the description isn't precise enough. And it's perfectly possible that current tooling isn't aggressive enough about pruning detail that might tempt the agent to ignore relevant files.

bburtenshaw 8mo ago

thanks for sharing the work. correct, we're currently working on evals for skills so you can compare skills between models and harnesses. we wrote a blog on getting agents to write CUDA kernels and evaluating them: https://huggingface.co/blog/upskill https://huggingface.co/blog/upskill

oofbey 8mo ago

This is a neat idea for a test. But the test is badly executed. A single comparison could just be a fluke. Compare it on a dozen tasks, trying each task a dozen times. Then you get data which is believable.