4 ms·

I can only recommend to regularly measure how many tokens a harness+model combination uses for a certain task There are huge token efficiency/bloat differences

by tosh 3mo ago

I can only recommend to regularly measure how many tokens a harness+model combination uses for a certain task There are huge token efficiency/bloat differences between agents while working on the same tasks, using the same model, in the same environment Yesterday I ran 10 agentic tasks using GPT 5.6 Sol in an ubuntu 26.04 vm a couple of times with different harnesses and got vastly different token usage. +-------------+-----------+-----------+-----------+-----------+--------+ | Harness | API total | Input | Cached | Uncached | Output | +-------------+-----------+-----------+-----------+-----------+--------+ | smol | 172,807 | 142,334 | 8,704 | 133,630 | 30,473 | | Pi | 427,211 | 392,767 | 137,216 | 255,551 | 34,444 | | OpenCode | 1,564,429 | 1,523,957 | 1,204,736 | 319,221 | 40,472 | | Codex | 3,005,744 | 2,953,154 | 2,649,344 | 303,810 | 52,590 | | Hermes | 3,856,611 | 3,808,231 | 3,167,232 | 640,999 | 48,380 | | Claude Code | 5,073,137 | 5,029,969 | 4,587,008 | 442,961 | 43,168 | +-------------+-----------+-----------+-----------+-----------+--------+ https://x.com/__tosh/status/2083593799872237680 https://x.com/__tosh/status/2083593799872237680 I'm not surprised that Claude Code is not optimized for an OpenAI model but I was still quite shocked re how much of a difference the harness makes. Disclaimer: I'm working on 'smol' which is a minimalist harness but it's really nothing special, just a minimal system prompt, no skills files, only tool is shell Do not underestimate how much popular harnesses are spamming the context window. The context window is very important.

esafak 3mo ago
Same task? How did the results compare?
- tosh 3mo ago
  the tasks were all simple agentic tasks like creating a checksum of a file, merging csvs and so on, fixing a makefile pipeline with known 'good' outcomes all harnesses could reach the outcomes, only cost, time, number of tool uses and so on were different (Claude Code failed once in 1 task but I think that was just an unfortunate outlier, the tasks aren't that difficult)
yojo 3mo ago
Claude Code injects a ton of tools into the system prompt, including their “memory system” that’s like 10k+ tokens. Depending on your task shape, this can easily double your task cost (e.g. a low context-using job that takes many turns, like a monitoring loop). You should use --disallowed-tools to prune any tools not needed for the task. Note that this is also a perpetual game of whack-a-mole since they’re always adding new tools.
- colourclash 3mo ago
  appreciate the tip, kind netizen
- visarga 3mo ago
  The system prompt part is surely cached across all users.
  - dcrazy 3mo ago
    For Anthropic models, yes. But OP was using Claude Code with GPT.
  - tosh 3mo ago
    the system prompt still takes up useful space in the context window and steers the model into unnecessary actions and over-thinking patterns
  - yojo 3mo ago
    You still pay cache token costs on API calls. Cache cost/token are 90% lower, but you pay it every single turn. I’m not sure if they let you skip the cache write cost on the first turn. That would imply cross-user caching infrastructure or special casing the default system prompt to give you a discount. Maybe? Away from the computer but you could try a “hello” in a fresh session and see what was billed.
    - cyanydeez 3mo ago
      Its inexpensive to reserve KV cache for the first turn and would benefit users, given the first turn already requires costly locating and allocating a model slot for a user. So yeah, when they banned tgird party harnesses there was a technical and $ case to have.
brettgriffin 3mo ago
Anything else I can read about smol? also, are you using a tool to collect those metrics? what is it?
- tosh 3mo ago
  smol is basically this 9 line python agent re-implemented in Go https://news.ycombinator.com/item?id=49006862 https://news.ycombinator.com/item?id=49006862 I'll have more about it in the next hours/days, you can follow me on twitter in the meantime (https://x.com/__tosh https://x.com/__tosh)
  - growt 3mo ago
    So smol is your agent? Do you have a GitHub link?
    - tosh 3mo ago
      coming in a few hours you can follow this org in the meantime https://github.com/smol-env https://github.com/smol-env or on twitter here: https://x.com/__tosh https://x.com/__tosh
    - tosh 3mo ago
      repo is published now: https://github.com/smol-env/smol https://github.com/smol-env/smol
azuanrb 3mo ago
What tools are you using to run the comparison? Or are you just running the prompts and checking them with something like ccusage? I’m asking because I’ve been looking for agent harness comparison tools too. I’m interested in more than just the inputs and outputs—I also want the system prompts, traces, and tool calls. It’s useful to understand why Codex, for example, uses more tokens while Pi doesn’t. Fewer tokens aren’t necessarily better if the agent skipped important checks. On the other hand, using more tokens could just mean it’s overthinking the process. Either way, seeing the full execution trace for the same task is really valuable.
- tosh 3mo ago
  I built an ad-hoc custom comparison framework to inspect system prompts, caching behaviour, tool call outputs, exact api requests and responses and so on I agree fewer tokens is not necessarily better but a bit counter-intuitively often the harness using fewer tokens is not only done faster but has better results (that said: of course check the results, look at the full traces, agree!)
  - azuanrb 3mo ago
    Thanks! I ended up built my own too. Just thought there are other better options out there that I might've missed.
Groxx 3mo ago
Running some local models and wondering what insanity was consuming 100k+ tokens to respond to "hello world" was rather eye-opening, yeah. Full of pointless fluff.
- cyanydeez 3mo ago
  I think prompt engineering went way to far into verbosity when at the same time agent and tool calling are much more important. Now a small system prompt can grab the contect it needs based on user context before bloating. On local models, i use dynamic context pruning.
bonestamp2 3mo ago
> Do not underestimate how much popular harnesses are spamming the context window. The context window is very important. Do we have any insight into whether it is actually spam and not useful info such as project or programming language specific context?
- tosh 3mo ago
  in my book anything that is (I'm sure well intentioned) and injected to help the agent — but doesn't help it — is a waste of tokens but even injected context that when I read it sounds useful can oversteer the model and make it second guess or take a more complicated route than it normally would (you can see this when looking at traces with and without that injected context) often harnesses also mention in their system prompt locations of markdown files that the model can consult if the model thinks they might help that hint alone as part of the system prompt can be strong enough to make the model read in more tokens than would have been necessary 'spam' is maybe a harsh way to say it unfortunately I don't see an easy way other than to invest time and tokens into finding out which parts of the added context (in system prompt, injected in turns etc etc) are actually helpful or harmful and when I'm just doing the easiest thing I could think of: start from nothing or close to nothing that seems to work better than what most harnesses are doing turns out GPT 5.6 Sol is all you need
  - mikodin 3mo ago
    I am super curious how much in your opinion the extra prompting is useful? >often harnesses also mention in their system prompt locations of markdown files that the model can consult if the model thinks they might help >that hint alone as part of the system prompt can be strong enough to make the model read in more tokens than would have been necessary I purposefully do this as I imagine it is useful. In my project I am seeing solid adherence to norms and a deep capacity to iterate on completed features. Essentially for each feature I have the model make a {featureName}.AGENTS.md at the root folder of the feature, where it maintains what is going on. I am moving between Claude Code and Codex atm, but I began this pattern when pi + kimi 2.6 was my main driver.
zuzululu 3mo ago
this benchmark is a very misleading because it doesn't necessarily point to these harnesses saving usage and much of the harness that comes with codex for example is designed at cross functional agentic usage which you will be leaving behind by using a different harness. openai spends a lot of time as does claude. also note that cached is what makes up bulk of the total and codex and claude has huge discounts in that scenario. the "spamming" is mostly compaction appending files, tool artifacts, images in its summary there is a known github issue for codex. only solution is periodic clean up but its also how a lot of the agentic orchestration is performed and able to work for days.
- tosh 3mo ago
  smol currently is very simple so it definitely does less things, like no subagent orchestration I will look into how token usage looks like for longer sessions and more complex tasks re caching: the cache ratio for this bench looks 'bad' for smol because it often finishes a task before caching kicks in (caching starts at 1024 tokens) thank you for flagging this
  - zuzululu 3mo ago
    sorry i didn't mean to call you out just wanted to point out at the comments thinking this means using sol/pi you can get many fold savings in tokens which isn't true smol looks interesting i think it could have a potential niche although tokens are only going to get cheaper and cheaper here
chrisweekly 3mo ago
Thanks for sharing. Naming tangent: FWIW I have recent interest in "smolvm" microvms from smolmachines.com, which are unrelated.
xienze 3mo ago
> There are huge token efficiency/bloat differences between agents while working on the same tasks, using the same model, in the same environment Be careful here. Remember these are non-deterministic models at the end of the day, and even with everything being "the same" you can have two runs where the same model, same harness, same tools can arrive at the same conclusion through a wildly different sequence of events.
- tosh 3mo ago
  agree, that makes it a bit tricky to compare (esp if you also want to add different models and reasoning levels into the mix) I will add more tasks (esp longer ones) and think more about grading, the current tasks were easy to grade because the desired outcomes are well specced but I will also look into more open ended tasks and how to grade those thank you!
ignore_prev 3mo ago
[dead]
applfanboysbgon 3mo ago
While there's a lot of bloat in the harnesses, almost all of it is cached. Your harness appears to be caching almost nothing. That will almost certainly bode extremely poorly for longer-running tasks than the simple ones you mentioned testing. Benchmaxxing for simplistic tasks could leave you not only suffering in completion rate on complex tasks but also possibly even not end up with the cost gains you're hoping for. And maybe that's fine if the only thing you use agents for is the very simple tasks, which is certainly a defensible decision, but probably bears mentioning when you're comparing it against harnesses with a different design goal.
- tosh 3mo ago
  smol is also prefix caching the uncached tokens are also from runs where smol finished a task below 1024 tokens (the minimum amount of tokens needed to activate caching) which is less tokens than other harnesses are using for their system prompt (!) > GPT-5.6 and later models: Caching is available for prefixes containing at least 1,024 tokens. This is a strict minimum. https://developers.openai.com/api/docs/guides/prompt-caching https://developers.openai.com/api/docs/guides/prompt-caching so in this specific case the count of uncached tokens for smol makes it look worse than it actually is that said: it does makes sense to add more tasks that are difficult enough to fill the context window to compare the harnesses for how well they deal with compaction staying below compaction (or with compaction at fewer compactions) is not only cheaper and faster, it also helps the agent stay on track
  - applfanboysbgon 3mo ago
    Yes, I do agree that avoiding compactions is extremely desirable, just make sure you're actually testing for how well that happens and the results!
    - tosh 3mo ago
      will do!
adoltech 3mo ago
> The context window is very important. It is everything. My experience with Claude Code is that you have to decide when to compact to make it efficient. It defaults everything to 1M context and it will never keep it in check. It is strange how little cache reads you hit in smol, that may be a configuration issue.
- nijave 3mo ago
  There's an env var you can set to tell it when to compact. It's especially helpful for models with big context that lose coherence much earlier.
- tosh 3mo ago
  I will look into it more to see if I have configured it wrong but I think the token efficiency hurts cache use as caching only starts at 1024 tokens so for tasks where smol is under or close to 1024 tokens most of them are uncached
bermudi 3mo ago
If you're working on smol? How complex is your work? The agent has to do everything using sed? Did you write your own tools? I guess my question is, why aren't you using pi? The difference in tokens between the two also makes super curious. The system prompt can't be that different (I'd even bet Pi's shorter) and the 4 tools shouldn't make as much of a difference. I'm gonna have to try it.
- tosh 3mo ago
  smol only has 1 tool: sh the system prompt of smol is shorter than the system prompt of Pi smol has no system prompt system prompt of Pi 0.83.0 """ You are an expert coding assistant operating inside pi, a coding agent harness. You help users by reading files, executing commands, editing code, and writing new files. Available tools: - read: Read file contents - bash: Execute bash commands (ls, grep, find, etc.) - edit: Make precise file edits with exact text replacement, including multiple disjoint edits in one call - write: Create or overwrite files In addition to the tools above, you may have access to other custom tools depending on the project. Guidelines: - Use bash for file operations like ls, rg, find - Use read to examine files instead of cat or sed. - Inspect PI_* environment variables for current model and session details. - Use edit for precise changes (edits[].oldText must match exactly) - When changing multiple separate locations in one file, use one edit call with multiple entries in edits[] instead of multiple edit calls - Each edits[].oldText is matched against the original file, not after earlier edits are applied. Do not emit overlapping or nested edits. Merge nearby changes into one edit. - Keep edits[].oldText as small as possible while still being unique in the file. Do not pad with large unchanged regions. - Use write only for new files or complete rewrites. - Be concise in your responses - Show file paths clearly when working with files Pi documentation (read only when the user asks about pi itself, its SDK, extensions, themes, skills, or TUI): - Main documentation: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/README.md - Additional docs: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/docs - Examples: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/examples (extensions, custom tools, SDK) - When reading pi docs or examples, resolve docs/... under Additional docs and examples/... under Examples, not the current working directory - When asked about: extensions (docs/extensions.md, examples/extensions/), themes (docs/themes.md), skills (docs/skills.md), prompt templates (docs/prompt-templates.md), TUI components (docs/tui.md), keybindings (docs/keybindings.md), SDK integrations (docs/sdk.md), custom providers (docs/custom-provider.md), adding models (docs/models.md), pi packages (docs/packages.md), environment variables (docs/environment-variables.md) - When working on pi topics, read the docs and examples, and follow .md cross-references before implementing - Always read pi .md files completely and follow links to related docs (e.g., tui.md for TUI API details) Current working directory: /workspace """
try-working 3mo ago
that's why i recommend running everything through a router.
skeptic_ai 3mo ago
I think it’s too easy to trick you into token counting by checking tokens used. I think we should only focus on Price, time and productivity. Some kind of ratio So a cheap model with many tokens or an expensive model with much less tokens could potentially be same price. Maybe only time will be the differential factor.
catlover76 3mo ago
[dead]

by tosh 3mo ago

esafak 3mo ago

Same task? How did the results compare?

tosh 3mo ago
the tasks were all simple agentic tasks like creating a checksum of a file, merging csvs and so on, fixing a makefile pipeline with known 'good' outcomes all harnesses could reach the outcomes, only cost, time, number of tool uses and so on were different (Claude Code failed once in 1 task but I think that was just an unfortunate outlier, the tasks aren't that difficult)

yojo 3mo ago

Claude Code injects a ton of tools into the system prompt, including their “memory system” that’s like 10k+ tokens. Depending on your task shape, this can easily double your task cost (e.g. a low context-using job that takes many turns, like a monitoring loop). You should use --disallowed-tools to prune any tools not needed for the task. Note that this is also a perpetual game of whack-a-mole since they’re always adding new tools.

colourclash 3mo ago
appreciate the tip, kind netizen
visarga 3mo ago
The system prompt part is surely cached across all users.
- dcrazy 3mo ago
  For Anthropic models, yes. But OP was using Claude Code with GPT.
- tosh 3mo ago
  the system prompt still takes up useful space in the context window and steers the model into unnecessary actions and over-thinking patterns
- yojo 3mo ago
  You still pay cache token costs on API calls. Cache cost/token are 90% lower, but you pay it every single turn. I’m not sure if they let you skip the cache write cost on the first turn. That would imply cross-user caching infrastructure or special casing the default system prompt to give you a discount. Maybe? Away from the computer but you could try a “hello” in a fresh session and see what was billed.
  - cyanydeez 3mo ago
    Its inexpensive to reserve KV cache for the first turn and would benefit users, given the first turn already requires costly locating and allocating a model slot for a user. So yeah, when they banned tgird party harnesses there was a technical and $ case to have.

brettgriffin 3mo ago

Anything else I can read about smol? also, are you using a tool to collect those metrics? what is it?

tosh 3mo ago
smol is basically this 9 line python agent re-implemented in Go https://news.ycombinator.com/item?id=49006862 https://news.ycombinator.com/item?id=49006862 I'll have more about it in the next hours/days, you can follow me on twitter in the meantime (https://x.com/__tosh https://x.com/__tosh)
- growt 3mo ago
  So smol is your agent? Do you have a GitHub link?
  - tosh 3mo ago
    coming in a few hours you can follow this org in the meantime https://github.com/smol-env https://github.com/smol-env or on twitter here: https://x.com/__tosh https://x.com/__tosh
  - tosh 3mo ago
    repo is published now: https://github.com/smol-env/smol https://github.com/smol-env/smol

azuanrb 3mo ago

What tools are you using to run the comparison? Or are you just running the prompts and checking them with something like ccusage? I’m asking because I’ve been looking for agent harness comparison tools too. I’m interested in more than just the inputs and outputs—I also want the system prompts, traces, and tool calls. It’s useful to understand why Codex, for example, uses more tokens while Pi doesn’t. Fewer tokens aren’t necessarily better if the agent skipped important checks. On the other hand, using more tokens could just mean it’s overthinking the process. Either way, seeing the full execution trace for the same task is really valuable.

tosh 3mo ago
I built an ad-hoc custom comparison framework to inspect system prompts, caching behaviour, tool call outputs, exact api requests and responses and so on I agree fewer tokens is not necessarily better but a bit counter-intuitively often the harness using fewer tokens is not only done faster but has better results (that said: of course check the results, look at the full traces, agree!)
- azuanrb 3mo ago
  Thanks! I ended up built my own too. Just thought there are other better options out there that I might've missed.

Groxx 3mo ago

Running some local models and wondering what insanity was consuming 100k+ tokens to respond to "hello world" was rather eye-opening, yeah. Full of pointless fluff.

cyanydeez 3mo ago
I think prompt engineering went way to far into verbosity when at the same time agent and tool calling are much more important. Now a small system prompt can grab the contect it needs based on user context before bloating. On local models, i use dynamic context pruning.

bonestamp2 3mo ago

> Do not underestimate how much popular harnesses are spamming the context window. The context window is very important. Do we have any insight into whether it is actually spam and not useful info such as project or programming language specific context?

tosh 3mo ago
in my book anything that is (I'm sure well intentioned) and injected to help the agent — but doesn't help it — is a waste of tokens but even injected context that when I read it sounds useful can oversteer the model and make it second guess or take a more complicated route than it normally would (you can see this when looking at traces with and without that injected context) often harnesses also mention in their system prompt locations of markdown files that the model can consult if the model thinks they might help that hint alone as part of the system prompt can be strong enough to make the model read in more tokens than would have been necessary 'spam' is maybe a harsh way to say it unfortunately I don't see an easy way other than to invest time and tokens into finding out which parts of the added context (in system prompt, injected in turns etc etc) are actually helpful or harmful and when I'm just doing the easiest thing I could think of: start from nothing or close to nothing that seems to work better than what most harnesses are doing turns out GPT 5.6 Sol is all you need
- mikodin 3mo ago
  I am super curious how much in your opinion the extra prompting is useful? >often harnesses also mention in their system prompt locations of markdown files that the model can consult if the model thinks they might help >that hint alone as part of the system prompt can be strong enough to make the model read in more tokens than would have been necessary I purposefully do this as I imagine it is useful. In my project I am seeing solid adherence to norms and a deep capacity to iterate on completed features. Essentially for each feature I have the model make a {featureName}.AGENTS.md at the root folder of the feature, where it maintains what is going on. I am moving between Claude Code and Codex atm, but I began this pattern when pi + kimi 2.6 was my main driver.

zuzululu 3mo ago

this benchmark is a very misleading because it doesn't necessarily point to these harnesses saving usage and much of the harness that comes with codex for example is designed at cross functional agentic usage which you will be leaving behind by using a different harness. openai spends a lot of time as does claude. also note that cached is what makes up bulk of the total and codex and claude has huge discounts in that scenario. the "spamming" is mostly compaction appending files, tool artifacts, images in its summary there is a known github issue for codex. only solution is periodic clean up but its also how a lot of the agentic orchestration is performed and able to work for days.

tosh 3mo ago
smol currently is very simple so it definitely does less things, like no subagent orchestration I will look into how token usage looks like for longer sessions and more complex tasks re caching: the cache ratio for this bench looks 'bad' for smol because it often finishes a task before caching kicks in (caching starts at 1024 tokens) thank you for flagging this
- zuzululu 3mo ago
  sorry i didn't mean to call you out just wanted to point out at the comments thinking this means using sol/pi you can get many fold savings in tokens which isn't true smol looks interesting i think it could have a potential niche although tokens are only going to get cheaper and cheaper here

chrisweekly 3mo ago

Thanks for sharing. Naming tangent: FWIW I have recent interest in "smolvm" microvms from smolmachines.com, which are unrelated.

xienze 3mo ago

> There are huge token efficiency/bloat differences between agents while working on the same tasks, using the same model, in the same environment Be careful here. Remember these are non-deterministic models at the end of the day, and even with everything being "the same" you can have two runs where the same model, same harness, same tools can arrive at the same conclusion through a wildly different sequence of events.

tosh 3mo ago
agree, that makes it a bit tricky to compare (esp if you also want to add different models and reasoning levels into the mix) I will add more tasks (esp longer ones) and think more about grading, the current tasks were easy to grade because the desired outcomes are well specced but I will also look into more open ended tasks and how to grade those thank you!

ignore_prev 3mo ago

[dead]

applfanboysbgon 3mo ago

While there's a lot of bloat in the harnesses, almost all of it is cached. Your harness appears to be caching almost nothing. That will almost certainly bode extremely poorly for longer-running tasks than the simple ones you mentioned testing. Benchmaxxing for simplistic tasks could leave you not only suffering in completion rate on complex tasks but also possibly even not end up with the cost gains you're hoping for. And maybe that's fine if the only thing you use agents for is the very simple tasks, which is certainly a defensible decision, but probably bears mentioning when you're comparing it against harnesses with a different design goal.

tosh 3mo ago
smol is also prefix caching the uncached tokens are also from runs where smol finished a task below 1024 tokens (the minimum amount of tokens needed to activate caching) which is less tokens than other harnesses are using for their system prompt (!) > GPT-5.6 and later models: Caching is available for prefixes containing at least 1,024 tokens. This is a strict minimum. https://developers.openai.com/api/docs/guides/prompt-caching https://developers.openai.com/api/docs/guides/prompt-caching so in this specific case the count of uncached tokens for smol makes it look worse than it actually is that said: it does makes sense to add more tasks that are difficult enough to fill the context window to compare the harnesses for how well they deal with compaction staying below compaction (or with compaction at fewer compactions) is not only cheaper and faster, it also helps the agent stay on track
- applfanboysbgon 3mo ago
  Yes, I do agree that avoiding compactions is extremely desirable, just make sure you're actually testing for how well that happens and the results!
  - tosh 3mo ago
    will do!

adoltech 3mo ago

> The context window is very important. It is everything. My experience with Claude Code is that you have to decide when to compact to make it efficient. It defaults everything to 1M context and it will never keep it in check. It is strange how little cache reads you hit in smol, that may be a configuration issue.

nijave 3mo ago
There's an env var you can set to tell it when to compact. It's especially helpful for models with big context that lose coherence much earlier.
tosh 3mo ago
I will look into it more to see if I have configured it wrong but I think the token efficiency hurts cache use as caching only starts at 1024 tokens so for tasks where smol is under or close to 1024 tokens most of them are uncached

bermudi 3mo ago

If you're working on smol? How complex is your work? The agent has to do everything using sed? Did you write your own tools? I guess my question is, why aren't you using pi? The difference in tokens between the two also makes super curious. The system prompt can't be that different (I'd even bet Pi's shorter) and the 4 tools shouldn't make as much of a difference. I'm gonna have to try it.

tosh 3mo ago
smol only has 1 tool: sh the system prompt of smol is shorter than the system prompt of Pi smol has no system prompt system prompt of Pi 0.83.0 """ You are an expert coding assistant operating inside pi, a coding agent harness. You help users by reading files, executing commands, editing code, and writing new files. Available tools: - read: Read file contents - bash: Execute bash commands (ls, grep, find, etc.) - edit: Make precise file edits with exact text replacement, including multiple disjoint edits in one call - write: Create or overwrite files In addition to the tools above, you may have access to other custom tools depending on the project. Guidelines: - Use bash for file operations like ls, rg, find - Use read to examine files instead of cat or sed. - Inspect PI_* environment variables for current model and session details. - Use edit for precise changes (edits[].oldText must match exactly) - When changing multiple separate locations in one file, use one edit call with multiple entries in edits[] instead of multiple edit calls - Each edits[].oldText is matched against the original file, not after earlier edits are applied. Do not emit overlapping or nested edits. Merge nearby changes into one edit. - Keep edits[].oldText as small as possible while still being unique in the file. Do not pad with large unchanged regions. - Use write only for new files or complete rewrites. - Be concise in your responses - Show file paths clearly when working with files Pi documentation (read only when the user asks about pi itself, its SDK, extensions, themes, skills, or TUI): - Main documentation: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/README.md - Additional docs: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/docs - Examples: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/examples (extensions, custom tools, SDK) - When reading pi docs or examples, resolve docs/... under Additional docs and examples/... under Examples, not the current working directory - When asked about: extensions (docs/extensions.md, examples/extensions/), themes (docs/themes.md), skills (docs/skills.md), prompt templates (docs/prompt-templates.md), TUI components (docs/tui.md), keybindings (docs/keybindings.md), SDK integrations (docs/sdk.md), custom providers (docs/custom-provider.md), adding models (docs/models.md), pi packages (docs/packages.md), environment variables (docs/environment-variables.md) - When working on pi topics, read the docs and examples, and follow .md cross-references before implementing - Always read pi .md files completely and follow links to related docs (e.g., tui.md for TUI API details) Current working directory: /workspace """

try-working 3mo ago

that's why i recommend running everything through a router.

skeptic_ai 3mo ago

I think it’s too easy to trick you into token counting by checking tokens used. I think we should only focus on Price, time and productivity. Some kind of ratio So a cheap model with many tokens or an expensive model with much less tokens could potentially be same price. Maybe only time will be the differential factor.

catlover76 3mo ago

[dead]