4 ms·

I am glad articles like this are finally starting to get some momentum around what I call the LLM magic box industry. From caveman mode to RTK to semantic searc

I am glad articles like this are finally starting to get some momentum around what I call the LLM magic box industry. From caveman mode to RTK to semantic search and everything in between. Developers have become magicians that cast spells instead of engineers. It sucks at work especially with everyone so sure that their magic spell is the one for ultimate token savings. My criteria are: if it’s not in a harness it’s probably not that good (the best ideas float up to Codex/Claude imo) and any GitHub advertising some percent of token savings is not to be trusted. It’s hard to avoid the snake oil and I hope people start thinking critically on this stuff.

blubber 3mo ago
There is a conflict of interest, though.
- baq 3mo ago
  Only in inference, but if you consider that they’re reinvesting inference performance in training I think the conflict argument is overblown.
arcanemachiner 3mo ago
The idea itself is sound: If you can reduce the signal-to-noise ratio in the context window, then that's a good thing. Whether or not RTK actually does this has not been established. I would be glad to see some proper benchmarks done on the actual difference this tool makes (not some meaningless "up to 90%" type of language).
- lackoftactics 3mo ago
  I was wondering if that impacts the accuracy, obviously the rtk output wasn't in the training dataset, but maybe it doesn't matter at the end
- celrod 3mo ago
  I found this, which has some: https://arxiv.org/pdf/2605.28876 https://arxiv.org/pdf/2605.28876 TLDR: RTK does not look good according to the author's benchmark.
  - deleted 3mo ago
    [deleted]
striking 3mo ago
I mean it kind of already is in harnesses. Codex and Claude Code both have subagent tools. You could probably get a similar token output cut just by asking Claude Code to run all commands with Haiku as a summarizing subagent.
lackoftactics 3mo ago
I have to say I made a similiar mistake with trusting semantic search is the next big thing. My opinion shifted, but it made sense for me for too long
philipbjorge 3mo ago
I'll go further and note that some of the optimizations I've seen in rtk for things like `git status` have actually bubbled up into the model layer -- Codex is regularly making tool calls like `git status --short` instead of `git status`.
AndyNemmity 3mo ago
This is why I Blind A/B test everything. I burn a ton of tokens, but things actually have to prove their value. And the vast majority of things do not come close to doing so. I have my own AI agent full of stuff. I blind A/B test everything, but I also don't think the results are all that useful as a signal to others. Just because I Blind A/B test it 4 months ago, it's maybe not meaningful today. Maybe the word choices I use dramatically impact things. I do it, because I can prove the value, and see it with my own eyes. I don't even bother publishing the specific Blind A/B tests. Also, I've seen other people try to Blind A/B test and get it very wrong. If your measurements aren't good, the test is meaningless. I don't know. We're all working on these problems together. There's a lot of black magic (which is why I rely on hooks a lot). I'm sure I have tons of black magic, I have a large little AI Agent. But what I know for certain, is it works for me. All it takes is for me to not use it, and I honestly don't know how everyone currently works with AI. I will link it, but it is not an endorsement for what you do. Mostly only other software engineers use it. And it's so very specific to the things I have to do. At best, maybe it sparks an idea for you to implement on your own. https://github.com/notque/vexjoy-agent https://github.com/notque/vexjoy-agent
deleted 3mo ago
[deleted]
tontinton 3mo ago
Totally wrong, you underestimate the frontier's incompetence in anything other than building LLM models (ehm ehm flickering TUI for a year "written like a game engine"). I ran a bunch of benchmarks and there are proven ways to reduce tokens while achieving the same results (finding the same CVEs / finding the same bugs in CRs, etc...). See https://maki.sh https://maki.sh, it's my own little proof.
- rescbr 3mo ago
  Maki is awesome. Thanks! I'm using it on my X220 and it flies in comparison to OpenCode et al.
  - tontinton 3mo ago
    Enjoy, I can't go back to other agents now, too spoiled by the speed
    - 0xc133 3mo ago
      I also have become a maki convert and I really like it. I ran into an issue with the dynamic model provider that I should probably make a patch for; list_models doesn’t use the `<provider> models` output at all but instead tries to look up `<provider> resolve`’s base URL + /v1/models, which breaks on a provider like Z.ai which doesn’t have /v1/ anywhere in the path…
      - tontinton 3mo ago
        Create an issue! :)
- lackoftactics 3mo ago
  What is your approach for reducing token usage and is it different than rtk?
  - tontinton 3mo ago
lopsotronic 3mo ago
Oh, this gold rush has breathed new life into the old school Semantic guys. Lord knows the DITA priesthood has been running low on rubes, so this new era is a godsend. Re-coding all of your org's content into a verbose granular schema, that's what will fix these AI things. It's going to give your LLM superpowers! Semantic superpowers! While everyone completely ignores the utter lack of coupling between the actual language and whatever nonsense is in the element / structure naming. Or the fact that every single thing has to go through some horrible 1990s era parser, which breaks constantly, and now everyone's shovelling the full markup into the very tiny confused mouth of the AI. Or that now everyone needs specialized software to display anything. Or the everything. My dudes, the thing you're trying to do with this stuff is already done in the vectorizations. You can use math for a lot of it now, instead of someone hand coding "poplar" as "tree" in a totally flat tree structure.
evilduck 3mo ago
>the best ideas float up to Codex/Claude imo They only float up if people create things like RTK and other people try them though. It's fair to sit this one out and let others figure out if it's worth it or not but tools like RTK, Headroom, caveman mode and others do reduce input and output tokens that need to be processed, and for local LLMs that can have measurable speedups. Whether or not that ultimately hurts the resulting output I don't have enough data to say, but I am happy to play with them to find out.
- chatmasta 3mo ago
  Also the incentives aren’t exactly aligned. Yes, Anthropic et. al want you to have efficient token usage (because you’ll use it more, and because of some competitive pressure). But it’s not their first priority especially when they make more money with more tokens. If a tool like rtk improves token efficiency, but has some negative impact on quality, should Anthropic integrate it immediately? Where is the line? This kind of decision is arguably better left to the user. What they should maybe do, is have a parameter similar to effort level, that allows the user to opt into native features for token minimizing. Make the tools available but leave the choice of the fidelity/savings tradeoff up to the user.
mingqiz 3mo ago
But Claude especially copy opensource ideas after they are widely used for months
Zababa 3mo ago
My criteria is "do they measure performance, or at least even try to?". Caveman [1], RTK [2] and more recently ponytail [3] don't or use a few trivial tests. Those projects don't measure performance on widely used benchmarks (like SWE Pro and stuff), that have their issues but at least it would give some indication. They also don't measure "big model + caveman vs smaller model". I've had a few times where removing all custom instructions that I started using with model N-2 made model N perform way better, so I'm very suspicious of everything that changes how the model works, it's easy to get degraded performance silently and suddenly you're paying latest Opus costs for 6 months old Sonnet performance. [1]: https://github.com/JuliusBrussee/caveman https://github.com/JuliusBrussee/caveman [2]: https://github.com/rtk-ai/rtk https://github.com/rtk-ai/rtk [3]: https://github.com/DietrichGebert/ponytail https://github.com/DietrichGebert/ponytail

blubber 3mo ago

There is a conflict of interest, though.

baq 3mo ago
Only in inference, but if you consider that they’re reinvesting inference performance in training I think the conflict argument is overblown.

arcanemachiner 3mo ago

The idea itself is sound: If you can reduce the signal-to-noise ratio in the context window, then that's a good thing. Whether or not RTK actually does this has not been established. I would be glad to see some proper benchmarks done on the actual difference this tool makes (not some meaningless "up to 90%" type of language).

lackoftactics 3mo ago
I was wondering if that impacts the accuracy, obviously the rtk output wasn't in the training dataset, but maybe it doesn't matter at the end
celrod 3mo ago
I found this, which has some: https://arxiv.org/pdf/2605.28876 https://arxiv.org/pdf/2605.28876 TLDR: RTK does not look good according to the author's benchmark.
- deleted 3mo ago
  [deleted]

striking 3mo ago

I mean it kind of already is in harnesses. Codex and Claude Code both have subagent tools. You could probably get a similar token output cut just by asking Claude Code to run all commands with Haiku as a summarizing subagent.

lackoftactics 3mo ago

I have to say I made a similiar mistake with trusting semantic search is the next big thing. My opinion shifted, but it made sense for me for too long

philipbjorge 3mo ago

I'll go further and note that some of the optimizations I've seen in rtk for things like `git status` have actually bubbled up into the model layer -- Codex is regularly making tool calls like `git status --short` instead of `git status`.

AndyNemmity 3mo ago

This is why I Blind A/B test everything. I burn a ton of tokens, but things actually have to prove their value. And the vast majority of things do not come close to doing so. I have my own AI agent full of stuff. I blind A/B test everything, but I also don't think the results are all that useful as a signal to others. Just because I Blind A/B test it 4 months ago, it's maybe not meaningful today. Maybe the word choices I use dramatically impact things. I do it, because I can prove the value, and see it with my own eyes. I don't even bother publishing the specific Blind A/B tests. Also, I've seen other people try to Blind A/B test and get it very wrong. If your measurements aren't good, the test is meaningless. I don't know. We're all working on these problems together. There's a lot of black magic (which is why I rely on hooks a lot). I'm sure I have tons of black magic, I have a large little AI Agent. But what I know for certain, is it works for me. All it takes is for me to not use it, and I honestly don't know how everyone currently works with AI. I will link it, but it is not an endorsement for what you do. Mostly only other software engineers use it. And it's so very specific to the things I have to do. At best, maybe it sparks an idea for you to implement on your own. https://github.com/notque/vexjoy-agent https://github.com/notque/vexjoy-agent

deleted 3mo ago

[deleted]

tontinton 3mo ago

Totally wrong, you underestimate the frontier's incompetence in anything other than building LLM models (ehm ehm flickering TUI for a year "written like a game engine"). I ran a bunch of benchmarks and there are proven ways to reduce tokens while achieving the same results (finding the same CVEs / finding the same bugs in CRs, etc...). See https://maki.sh https://maki.sh, it's my own little proof.

rescbr 3mo ago
Maki is awesome. Thanks! I'm using it on my X220 and it flies in comparison to OpenCode et al.
- tontinton 3mo ago
  Enjoy, I can't go back to other agents now, too spoiled by the speed
  - 0xc133 3mo ago
    I also have become a maki convert and I really like it. I ran into an issue with the dynamic model provider that I should probably make a patch for; list_models doesn’t use the `<provider> models` output at all but instead tries to look up `<provider> resolve`’s base URL + /v1/models, which breaks on a provider like Z.ai which doesn’t have /v1/ anywhere in the path…
    - tontinton 3mo ago
      Create an issue! :)
lackoftactics 3mo ago
What is your approach for reducing token usage and is it different than rtk?
- tontinton 3mo ago

lopsotronic 3mo ago

Oh, this gold rush has breathed new life into the old school Semantic guys. Lord knows the DITA priesthood has been running low on rubes, so this new era is a godsend. Re-coding all of your org's content into a verbose granular schema, that's what will fix these AI things. It's going to give your LLM superpowers! Semantic superpowers! While everyone completely ignores the utter lack of coupling between the actual language and whatever nonsense is in the element / structure naming. Or the fact that every single thing has to go through some horrible 1990s era parser, which breaks constantly, and now everyone's shovelling the full markup into the very tiny confused mouth of the AI. Or that now everyone needs specialized software to display anything. Or the everything. My dudes, the thing you're trying to do with this stuff is already done in the vectorizations. You can use math for a lot of it now, instead of someone hand coding "poplar" as "tree" in a totally flat tree structure.

evilduck 3mo ago

>the best ideas float up to Codex/Claude imo They only float up if people create things like RTK and other people try them though. It's fair to sit this one out and let others figure out if it's worth it or not but tools like RTK, Headroom, caveman mode and others do reduce input and output tokens that need to be processed, and for local LLMs that can have measurable speedups. Whether or not that ultimately hurts the resulting output I don't have enough data to say, but I am happy to play with them to find out.

chatmasta 3mo ago
Also the incentives aren’t exactly aligned. Yes, Anthropic et. al want you to have efficient token usage (because you’ll use it more, and because of some competitive pressure). But it’s not their first priority especially when they make more money with more tokens. If a tool like rtk improves token efficiency, but has some negative impact on quality, should Anthropic integrate it immediately? Where is the line? This kind of decision is arguably better left to the user. What they should maybe do, is have a parameter similar to effort level, that allows the user to opt into native features for token minimizing. Make the tools available but leave the choice of the fidelity/savings tradeoff up to the user.

mingqiz 3mo ago

But Claude especially copy opensource ideas after they are widely used for months

Zababa 3mo ago

My criteria is "do they measure performance, or at least even try to?". Caveman [1], RTK [2] and more recently ponytail [3] don't or use a few trivial tests. Those projects don't measure performance on widely used benchmarks (like SWE Pro and stuff), that have their issues but at least it would give some indication. They also don't measure "big model + caveman vs smaller model". I've had a few times where removing all custom instructions that I started using with model N-2 made model N perform way better, so I'm very suspicious of everything that changes how the model works, it's easy to get degraded performance silently and suddenly you're paying latest Opus costs for 6 months old Sonnet performance. [1]: https://github.com/JuliusBrussee/caveman https://github.com/JuliusBrussee/caveman [2]: https://github.com/rtk-ai/rtk https://github.com/rtk-ai/rtk [3]: https://github.com/DietrichGebert/ponytail https://github.com/DietrichGebert/ponytail