3 ms·

Don't use the default harness, write your own instead.

by Muromec 3mo ago

Supermancho 3mo ago
This is the way. Making your own agent to have a sticky memory context that is prepended to every execution is necessary to ensure each task is bounded by those precepts.
- Muromec 3mo ago
  The trick I'm doing -- the model is given a tool that runs a prompt in the current thread to consolidate it's working memory and identity (it has a memory tool bound to the agent persona). When the prompt ends, the parts of memory that are marked as identity are merged together into a new system prompt, then the context restarts with only system prompt and this tool call surviving. Then it just keeps going.
  - mblangie 3mo ago
    [flagged]
- drob518 3mo ago
  Doesn’t that blow your token cache hit rate and balloon your costs (essentially everything is billed at the input token rate, not the cached rate)?
  - haus20xx 3mo ago
    If it moved a block like <system_prompt> <tools_etc> <project_prompts> <user_call> <llm_response> => <system_prompt> <tools_etc> //Removed project_prompts <user_call> <llm_response> <user_call> <project_prompts> //reinserted <llm_response> Cache would break but if you did instead <system_prompt> <tools_etc> <project_prompts> <user_call> <llm_response> <user_call> <project_prompts> //Duplicated with new user messsage <llm_response> It wouldn't bust cache, it would just make your input prompts slightly larger. Technically inefficient as you're duplicating the same rules over and over but I imagine for a smallish checklist/principles that it is tremendously more efficient than a cache break every message
    - cyanydeez 3mo ago
      so first, we treat context like a stack. Things are popped onto the stack; if we try to remove anything but the top of the stack, we'll break the cache. So a good harness should be popping tool prompts, user hints after every return from the top so the context of tools and user hints never repeats and are allways prefixed to the current user prompt. I've never inspected any harness, but opencode with the dynamic context compaction plugin appears to do this well.
    - Muromec 3mo ago
      By the time it does 10 turns of tool calls it forgets about the nagging again. It can and does ignore direct instruction from the user prompt too, sometimes repeatedly
TeMPOraL 3mo ago
Any way to do that AND use subscription instead of per-token pricing from SOTA providers?
- Muromec 3mo ago
  If the harness runs on your localhost, but the inference doesn't, it usually means it's calling some API. Whether you want to break TOS of your provider like that or simply buy tokens from our friends from UTC+8 timezone is an open question.

Don't use the default harness, write your own instead.

by Muromec 3mo ago

Don't use the default harness, write your own instead.

This is the way. Making your own agent to have a sticky memory context that is prepended to every execution is necessary to ensure each task is bounded by those precepts.

Muromec 3mo ago
The trick I'm doing -- the model is given a tool that runs a prompt in the current thread to consolidate it's working memory and identity (it has a memory tool bound to the agent persona). When the prompt ends, the parts of memory that are marked as identity are merged together into a new system prompt, then the context restarts with only system prompt and this tool call surviving. Then it just keeps going.
- mblangie 3mo ago
  [flagged]
drob518 3mo ago
Doesn’t that blow your token cache hit rate and balloon your costs (essentially everything is billed at the input token rate, not the cached rate)?
- haus20xx 3mo ago
  If it moved a block like <system_prompt> <tools_etc> <project_prompts> <user_call> <llm_response> => <system_prompt> <tools_etc> //Removed project_prompts <user_call> <llm_response> <user_call> <project_prompts> //reinserted <llm_response> Cache would break but if you did instead <system_prompt> <tools_etc> <project_prompts> <user_call> <llm_response> <user_call> <project_prompts> //Duplicated with new user messsage <llm_response> It wouldn't bust cache, it would just make your input prompts slightly larger. Technically inefficient as you're duplicating the same rules over and over but I imagine for a smallish checklist/principles that it is tremendously more efficient than a cache break every message
  - cyanydeez 3mo ago
    so first, we treat context like a stack. Things are popped onto the stack; if we try to remove anything but the top of the stack, we'll break the cache. So a good harness should be popping tool prompts, user hints after every return from the top so the context of tools and user hints never repeats and are allways prefixed to the current user prompt. I've never inspected any harness, but opencode with the dynamic context compaction plugin appears to do this well.
  - Muromec 3mo ago
    By the time it does 10 turns of tool calls it forgets about the nagging again. It can and does ignore direct instruction from the user prompt too, sometimes repeatedly

TeMPOraL 3mo ago

Any way to do that AND use subscription instead of per-token pricing from SOTA providers?

Muromec 3mo ago
If the harness runs on your localhost, but the inference doesn't, it usually means it's calling some API. Whether you want to break TOS of your provider like that or simply buy tokens from our friends from UTC+8 timezone is an open question.