6 ms·

He's also missed a major step, which is to feed your skill into the LLM and ask it to critique it - after all, it's the LLM that's going to act on it, so asking

He's also missed a major step, which is to feed your skill into the LLM and ask it to critique it - after all, it's the LLM that's going to act on it, so asking it to assess first is kinda important. I've done that for his skills, here's the assessment: ========== Bottom line Against the agentskills.io guidance, they look more like workflow specs than polished agent skills. The largest gap is not correctness. It is skill design discipline: # stronger descriptions, # lighter defaults, # less mandatory process, # better degraded-mode handling, # clearer evidence that the skills were refined through trigger/output evals. Skill Score/10 write-a-prd 5.4 prd-to-issues 6.8 issues-to-tasks 6.0 code-review 7.6 final-audit 6.3 ========== LLM metaprogramming is extremely important, I've just finished a LLM-assisted design doc authoring session where the recommendations of the LLM are "Don't use a LLM for that part, it won't be reliable enough".

fergonco 6mo ago
> "Don't use a LLM for that part, it won't be reliable enough". You should now ask if the LLM is reliable enough when it says that. Jokes aside, how is this a major step he is missing? He is using those skills to be more efficient. How important is going against agentskills.io guidance?
- crustycoder 6mo ago
  Because he's asking the LLM to interpret those instructions to drive his process. If the skills are poorly defined or incomplete then the process will be as well, and the LLM may misinterpret, choose to ignore, or add its own parts. Skills are just another kind of programming, albeit at a pretty abstract level. A good initial review process for a Skill is to ask the LLM what it thinks the Skill means and where it thinks there are holes. Just writing it and then running it isn't sufficient. Another tip is to give the Skill the same input in multiple new sessions - to stop state carryover - collect the output from each session and then feed it back into the LLM and ask it to assess where and why the output was different.
  - hansmayer 6mo ago
    Oh dear, I thought you were merely sarcastic in your first comment. But you seem to have been fully converted to the LLM-religion, and actually believe they actually "think" or "know" anything?
    - crustycoder 6mo ago
      People have applied "think" to the actions of software for decades. Of course it LLM's don't "think" in the human sense, but "What the output of the model indicates in an approximate way about its current internal state" is a bit long winded...
      - hansmayer 6mo ago
        Maybe people who dont understand technology did, I can see that - my granpa also thought the computer was thinking when the windows hourglass showed up. Today maybe its the case again with the folks who dont know anything about it - you know that meme - ChatGPT always gives me correct answers for the domains I am not an expert in!
jddj 6mo ago
Is your premise here that LLMs have a unique or enhanced insight into how LLMs work best?
- crustycoder 6mo ago
  I wouldn't go that far but the only way I've found so far of getting a reasonable insight into why a LLM has chosen to do something is to ask it.
- alexwebb2 6mo ago
  Not OP but I’d back that assertion. When the model that’s interpreting it is the same model that’s going to be executing it, they share the same latent space state at the outset. So this is essentially asking whether models are able to answer questions about context they’re given, and of course the answer is yes.
  - didgeoridoo 6mo ago
    There is no evidence of this. Evals are quite different from "self-evals". The only robust way of determining if LLM instructions are "good" is to run them through the intended model lots of times and see if you consistently get the result you want. Asking the model if the instructions are good shows a very deep misunderstanding of how LLMs work.
    - crustycoder 6mo ago
      Is that based on your "deep understanding" of how LLMs work or have you actually tried it? If you watch the execution trace of a Skill in action, you can see that it's doing exactly this inspection when the skill runs - how could it possibly work any other way? Skills are just textual instructions, LLMs are perfectly capable of spotting inconsistencies, gaps and contradictions in them. Is that sufficient to create a good skill? No, of course not, you need to actually test them. To use an analogy, asking a LLM to critique a skill is like running lint on C code first to pick up egregious problems, running testcases is vital.
mpalmer 6mo ago
LLMs do not have special or unique insight into how best to prompt them. Not in the slightest. https://aphyr.com/posts/411-the-future-of-everything-is-lies-i-guess#unreliable-narrators https://aphyr.com/posts/411-the-future-of-everything-is-lies...
- crustycoder 6mo ago
  "Not in the slightest" is an overreach, the paper the second level down from that link doesn't really support the conclusion in the blog post - the paper is much more nuanced. Are they going to fib to you sometimes? Yes of course, but that doesn't mean there's no value in behavioural metaqueries. Like most new tech, the discussion tends to polarise into "Best thing evah!" and "Utter shite!" The truth is somewhere in between.
  - mpalmer 6mo ago
    You're retreating from your position. You started at "major step" and "extremely important", and you've arrived at "there's not no value".
    - crustycoder 6mo ago
      Picking phrases from what I said and deliberately misquoting them out of context does not make you right.
      - mpalmer 6mo ago
        How exactly did I misquote you?
        crustycoder 6mo ago
        Go figure it out, it will be a useful challenge for you.
  - hansmayer 6mo ago
    > Like most new tech It's nothing like "most new tech". Most new tech tends to be adopted early by young people and experienced techies. In this case it is mostly the opposite: The teens absolutely hate it, probably because the shitty AI content does not inspire the young mind, and the experienced techies see it for what it is. I've never seen such "new tech" which was cheered on by the proverbial average "boomers" (i.e. old people doing "office jobs", not the literal age bracket) and despised by the young folks and experienced experts of all ages.
swingboy 6mo ago
Do these scores actually mean anything? Isn’t the LLM just making up something? If you ran the exact same prompt through 10 times would you get those same scores every single time?
- grey-area 6mo ago
  Yes I'd be interested in that answer too - these scores are most likely just generated in an arbitrary way, given how LLMs work. Given how they work in generating text it didn't actually keep a score and add to it each time it found a plus point in the skill as a human might in evaluating something. At this point I'd discount most advice given by people using LLMs, because most of them don't recognise the inadequacies and failure modes of these machines (like the OP here) and just assume that because output is superficially convincing it is correct and based on something. Do these skills meaningfully improve performance? Should we even need them when interacting with LLMs?
  - crustycoder 6mo ago
    They aren't arbitrary, as I said earlier I got the LLM to de a detailed analysis first, then summarise. If I was doing this "properly" for something I was doing myself I'd go through the LLM summary point by point and challenge anything I didn't think was right and fix things in the skill where I thought it was correct. You aren't going to have much success with LLMs if you don't understand that their primary goal is to produce plausible and coherent responses rather than ones that are necessarily correct (although they may be - hopefully). And yes, Skills *do* make a significant difference to performance, in exactly the same way that well written prompts do - because that's all they really are. If you just throw something at a LLM and tell it "do something with this" it will, but it probably won't be what you want and it will probably be different each time you ask. https://agentskills.io/home https://agentskills.io/home
    - grey-area 6mo ago
      It would be interesting to see one of these evals and how it generated the score, to work out whether it is in fact arbitrary or based on some scale of points. I found the summary above devoid of useful advice, what did you see as useful advice in it? > if you don't understand that their primary goal is to produce plausible and coherent responses rather than ones that are necessarily correct (although they may be - hopefully). If you really believe this you should perhaps re-evaluate the trust you appear to place in the conclusions of LLMs, particularly about their own workings and what makes a good skill or prompt for them.
grey-area 6mo ago
What does this even mean? It looks like typical LLM bloviation to me: 'skill design discipline', 'stronger descriptions' and 'lighter defaults'??!? This is meaningless pablum masquerading as advice. What specifically would this cause you to actually do to improve the skills in question? How would you measure that improvement in a non hand-wavy way? What do these scores mean and how were they calculated? Or perhaps you would ask your LLM how it would improve these skills? It will of course some up with some changes, but are they the right changes and how would you know?
- skydhash 6mo ago
  It’s all vibes based, we are not trying to be scientific here. /s I discard most LLM advice and skills because either a script is better (as the work is routine enough) or it could be expressed better with bullet points (generating tickets).
- hansmayer 6mo ago
  Great points, but I imagine it's a bit too heavy on the rigorousness requirement for the LLM crowd. The folks are high on this stuff and I am beginning to notice it's like trying to get a heavy pothead or crackhead of off their stuff. Don't you see it - if you just wave your hands a lot, and tell the LLM to be serious about it, the scores will just appear :) It's true in their own frame of reference.
- crustycoder 6mo ago
  I'm not going to repeat myself, I've already explained the context to you - funny how you seem to have ignored that. If you want to find out, do the experiment yourself.
hansmayer 6mo ago
You gotta love the randomly assigned score, like if LLM is actually able to measure anything. But then again, we now call a blob of text a "skill", so I guess it matches the overall bullshit pattern.
threecheese 6mo ago
Go even further, and add this into the skill-creator skill, and let the agent improve the skill regularly. I do this with determinism, and have my skills try to identify steps which can be scripted.

fergonco 6mo ago

> "Don't use a LLM for that part, it won't be reliable enough". You should now ask if the LLM is reliable enough when it says that. Jokes aside, how is this a major step he is missing? He is using those skills to be more efficient. How important is going against agentskills.io guidance?

crustycoder 6mo ago
Because he's asking the LLM to interpret those instructions to drive his process. If the skills are poorly defined or incomplete then the process will be as well, and the LLM may misinterpret, choose to ignore, or add its own parts. Skills are just another kind of programming, albeit at a pretty abstract level. A good initial review process for a Skill is to ask the LLM what it thinks the Skill means and where it thinks there are holes. Just writing it and then running it isn't sufficient. Another tip is to give the Skill the same input in multiple new sessions - to stop state carryover - collect the output from each session and then feed it back into the LLM and ask it to assess where and why the output was different.
- hansmayer 6mo ago
  Oh dear, I thought you were merely sarcastic in your first comment. But you seem to have been fully converted to the LLM-religion, and actually believe they actually "think" or "know" anything?
  - crustycoder 6mo ago
    People have applied "think" to the actions of software for decades. Of course it LLM's don't "think" in the human sense, but "What the output of the model indicates in an approximate way about its current internal state" is a bit long winded...
    - hansmayer 6mo ago
      Maybe people who dont understand technology did, I can see that - my granpa also thought the computer was thinking when the windows hourglass showed up. Today maybe its the case again with the folks who dont know anything about it - you know that meme - ChatGPT always gives me correct answers for the domains I am not an expert in!

jddj 6mo ago

Is your premise here that LLMs have a unique or enhanced insight into how LLMs work best?

crustycoder 6mo ago
I wouldn't go that far but the only way I've found so far of getting a reasonable insight into why a LLM has chosen to do something is to ask it.
alexwebb2 6mo ago
Not OP but I’d back that assertion. When the model that’s interpreting it is the same model that’s going to be executing it, they share the same latent space state at the outset. So this is essentially asking whether models are able to answer questions about context they’re given, and of course the answer is yes.
- didgeoridoo 6mo ago
  There is no evidence of this. Evals are quite different from "self-evals". The only robust way of determining if LLM instructions are "good" is to run them through the intended model lots of times and see if you consistently get the result you want. Asking the model if the instructions are good shows a very deep misunderstanding of how LLMs work.
  - crustycoder 6mo ago
    Is that based on your "deep understanding" of how LLMs work or have you actually tried it? If you watch the execution trace of a Skill in action, you can see that it's doing exactly this inspection when the skill runs - how could it possibly work any other way? Skills are just textual instructions, LLMs are perfectly capable of spotting inconsistencies, gaps and contradictions in them. Is that sufficient to create a good skill? No, of course not, you need to actually test them. To use an analogy, asking a LLM to critique a skill is like running lint on C code first to pick up egregious problems, running testcases is vital.

mpalmer 6mo ago

LLMs do not have special or unique insight into how best to prompt them. Not in the slightest. https://aphyr.com/posts/411-the-future-of-everything-is-lies-i-guess#unreliable-narrators https://aphyr.com/posts/411-the-future-of-everything-is-lies...

crustycoder 6mo ago
"Not in the slightest" is an overreach, the paper the second level down from that link doesn't really support the conclusion in the blog post - the paper is much more nuanced. Are they going to fib to you sometimes? Yes of course, but that doesn't mean there's no value in behavioural metaqueries. Like most new tech, the discussion tends to polarise into "Best thing evah!" and "Utter shite!" The truth is somewhere in between.
- mpalmer 6mo ago
  You're retreating from your position. You started at "major step" and "extremely important", and you've arrived at "there's not no value".
  - crustycoder 6mo ago
    Picking phrases from what I said and deliberately misquoting them out of context does not make you right.
    - mpalmer 6mo ago
      How exactly did I misquote you?
      - crustycoder 6mo ago
        Go figure it out, it will be a useful challenge for you.
- hansmayer 6mo ago
  > Like most new tech It's nothing like "most new tech". Most new tech tends to be adopted early by young people and experienced techies. In this case it is mostly the opposite: The teens absolutely hate it, probably because the shitty AI content does not inspire the young mind, and the experienced techies see it for what it is. I've never seen such "new tech" which was cheered on by the proverbial average "boomers" (i.e. old people doing "office jobs", not the literal age bracket) and despised by the young folks and experienced experts of all ages.

swingboy 6mo ago

Do these scores actually mean anything? Isn’t the LLM just making up something? If you ran the exact same prompt through 10 times would you get those same scores every single time?

grey-area 6mo ago
Yes I'd be interested in that answer too - these scores are most likely just generated in an arbitrary way, given how LLMs work. Given how they work in generating text it didn't actually keep a score and add to it each time it found a plus point in the skill as a human might in evaluating something. At this point I'd discount most advice given by people using LLMs, because most of them don't recognise the inadequacies and failure modes of these machines (like the OP here) and just assume that because output is superficially convincing it is correct and based on something. Do these skills meaningfully improve performance? Should we even need them when interacting with LLMs?
- crustycoder 6mo ago
  They aren't arbitrary, as I said earlier I got the LLM to de a detailed analysis first, then summarise. If I was doing this "properly" for something I was doing myself I'd go through the LLM summary point by point and challenge anything I didn't think was right and fix things in the skill where I thought it was correct. You aren't going to have much success with LLMs if you don't understand that their primary goal is to produce plausible and coherent responses rather than ones that are necessarily correct (although they may be - hopefully). And yes, Skills *do* make a significant difference to performance, in exactly the same way that well written prompts do - because that's all they really are. If you just throw something at a LLM and tell it "do something with this" it will, but it probably won't be what you want and it will probably be different each time you ask. https://agentskills.io/home https://agentskills.io/home
  - grey-area 6mo ago
    It would be interesting to see one of these evals and how it generated the score, to work out whether it is in fact arbitrary or based on some scale of points. I found the summary above devoid of useful advice, what did you see as useful advice in it? > if you don't understand that their primary goal is to produce plausible and coherent responses rather than ones that are necessarily correct (although they may be - hopefully). If you really believe this you should perhaps re-evaluate the trust you appear to place in the conclusions of LLMs, particularly about their own workings and what makes a good skill or prompt for them.

grey-area 6mo ago

What does this even mean? It looks like typical LLM bloviation to me: 'skill design discipline', 'stronger descriptions' and 'lighter defaults'??!? This is meaningless pablum masquerading as advice. What specifically would this cause you to actually do to improve the skills in question? How would you measure that improvement in a non hand-wavy way? What do these scores mean and how were they calculated? Or perhaps you would ask your LLM how it would improve these skills? It will of course some up with some changes, but are they the right changes and how would you know?

skydhash 6mo ago
It’s all vibes based, we are not trying to be scientific here. /s I discard most LLM advice and skills because either a script is better (as the work is routine enough) or it could be expressed better with bullet points (generating tickets).
hansmayer 6mo ago
Great points, but I imagine it's a bit too heavy on the rigorousness requirement for the LLM crowd. The folks are high on this stuff and I am beginning to notice it's like trying to get a heavy pothead or crackhead of off their stuff. Don't you see it - if you just wave your hands a lot, and tell the LLM to be serious about it, the scores will just appear :) It's true in their own frame of reference.
crustycoder 6mo ago
I'm not going to repeat myself, I've already explained the context to you - funny how you seem to have ignored that. If you want to find out, do the experiment yourself.

hansmayer 6mo ago

You gotta love the randomly assigned score, like if LLM is actually able to measure anything. But then again, we now call a blob of text a "skill", so I guess it matches the overall bullshit pattern.

threecheese 6mo ago

Go even further, and add this into the skill-creator skill, and let the agent improve the skill regularly. I do this with determinism, and have my skills try to identify steps which can be scripted.