4 ms·

Secret: "compile" that orchestration prompt. Determinism is solved by turning prompts into code that can in turn run agents or run code or both. Everyone misse

by fny 5mo ago

Secret: "compile" that orchestration prompt. Determinism is solved by turning prompts into code that can in turn run agents or run code or both. Everyone misses this pattern with skills: you can just drop code alongside a SKILL.md to guarantee certain behaviors, but for some reason everyone's addicted to writing prompts. You don't even need to build a CLI. A simple skill.py with tasks does it. You can even have helpers that call `claude -p`!

krzyk 5mo ago
Could you elaborate what does "compiling orchestration prompt" mean?
- throawayonthe 5mo ago
  a guess but i think they mean take the orchestration prompt and prompt yet another llm to turn that prompt into code..?
- Frost1x 5mo ago
  When you get some abstraction working you concretize it in something deterministic, or sort of “cache” that knowledge bit (aka write me a function, class, library, whatever). In the future, the nondeterministic path now has a deterministic piece to lean on as it explores the problem space. Rinse, repeat, eventually you have a mostly deterministic system now. Leave flexibility in space where you need that nondeterminism.
- LikesPwsh 5mo ago
  Rather than telling the LLM "loop through these files", tell it "write a script to loop through these files", then hard-code that script somewhere.
  - whattheheckheck 5mo ago
    The models will eventually be able to know that they need to do that to get the thing done from natural language
    - suttontom 5mo ago
      "The models will eventually..." Yeah but they haven't, and it's been years now. Also who cares? We have problems right now that need to be solved.
  - renticulous 5mo ago
    First we gave LLMs access to bash commands. Now we give them access to customized commands which they can reuse. It's English language extending its claws into deterministic programming language. Now can we please have backtracking and dynamic programming like thinking loop built into English language or such orchestration prompts.
- deleted 5mo ago
  [deleted]
robinduckett 5mo ago
Exactly this, I tend to work this way. I built an ingestion pipeline to pull concepts out of a novel using Qwen and push them into falkordb this way
Cyan488 5mo ago
What about when the model trusts itself more than the "black box" you gave it, and hallucinates its use or non-use in favor of reimplementation? I found this video about "intelligent disobedience" interesting. https://www.youtube.com/watch?v=Qu-00j9XuF0 https://www.youtube.com/watch?v=Qu-00j9XuF0
AndyNemmity 5mo ago
Yeah, that's how I do skills. If I can make a script, I do. Everything that can be deterministic should be. https://github.com/notque/vexjoy-agent https://github.com/notque/vexjoy-agent

by fny 5mo ago

krzyk 5mo ago

Could you elaborate what does "compiling orchestration prompt" mean?

throawayonthe 5mo ago
a guess but i think they mean take the orchestration prompt and prompt yet another llm to turn that prompt into code..?
Frost1x 5mo ago
When you get some abstraction working you concretize it in something deterministic, or sort of “cache” that knowledge bit (aka write me a function, class, library, whatever). In the future, the nondeterministic path now has a deterministic piece to lean on as it explores the problem space. Rinse, repeat, eventually you have a mostly deterministic system now. Leave flexibility in space where you need that nondeterminism.
LikesPwsh 5mo ago
Rather than telling the LLM "loop through these files", tell it "write a script to loop through these files", then hard-code that script somewhere.
- whattheheckheck 5mo ago
  The models will eventually be able to know that they need to do that to get the thing done from natural language
  - suttontom 5mo ago
    "The models will eventually..." Yeah but they haven't, and it's been years now. Also who cares? We have problems right now that need to be solved.
- renticulous 5mo ago
  First we gave LLMs access to bash commands. Now we give them access to customized commands which they can reuse. It's English language extending its claws into deterministic programming language. Now can we please have backtracking and dynamic programming like thinking loop built into English language or such orchestration prompts.
deleted 5mo ago
[deleted]

robinduckett 5mo ago

Exactly this, I tend to work this way. I built an ingestion pipeline to pull concepts out of a novel using Qwen and push them into falkordb this way

Cyan488 5mo ago

What about when the model trusts itself more than the "black box" you gave it, and hallucinates its use or non-use in favor of reimplementation? I found this video about "intelligent disobedience" interesting. https://www.youtube.com/watch?v=Qu-00j9XuF0 https://www.youtube.com/watch?v=Qu-00j9XuF0

AndyNemmity 5mo ago

Yeah, that's how I do skills. If I can make a script, I do. Everything that can be deterministic should be. https://github.com/notque/vexjoy-agent https://github.com/notque/vexjoy-agent