4 ms·

For me the sweet spot is somewhere under 500k depending on how extensive I want to get. You can build up a sizable effort project in half a million tokens with

For me the sweet spot is somewhere under 500k depending on how extensive I want to get. You can build up a sizable effort project in half a million tokens with Claude, with Claude having all the context from ground 0 to wherever you're off at.

cyanydeez 3mo ago
I'm always curious what you guys are working on; every git repo I've run a local model on and stick below <100k to increase speed seems effective enough to scope patches and changes.
- jdoe1337halo 3mo ago
  They are just talking to the model in CC, while staying in a single thread. Doubt they have any actual coding knowledge to compartmentalize different problems in the codebase.
  - giancarlostoro 3mo ago
    Depends on the programming language I'm using for a given project, and the domain I'm working with. I've been coding as a hobbyist for nearly two decades now (since my teens), professionally for 9 years, and was a TA before that for roughly 3 years at one of the best colleges for this field in the state (at least back then it was) where I taught other students about programming, in some cases I was their primary learning resource. But yeah, I have no idea about anything about software because you made an assumption off very little to go by.
- KronisLV 3mo ago
  My current Claude Code session has been going on for like 35 hours and has used up around 400 million tokens, thankfully almost all of those being cached (95-98%) - pretty typical for long form agentic work. First you spend like 2-3 hours working on a plan, once you have that you just tell the model to go and implement it, do adversarial sub-agent review loops before each commit and also make sure that all tooling and tests pass (including coverage requirements). You do need to poke it in a slightly different direction every few hours, though. Not even any novel work, just some refactoring and SSE notification hardening, bug fixes, alongside environment tuning and getting rid of some bottlenecks (also migrated from Oracle to PostgreSQL but that's mostly done). That said, Kimi somehow manages to use less context in the main thread than Anthropic's models (even when you use sub-agents and also dynamic workflows in Claude Code), might have something to do with either how the model is tuned or their Kimi Code harness - because even in most of the longer form sessions it doesn't seem to fill up quite as quickly (note: because the kimi vis tool doesn't have a full summary view across all agents, these are the main long running agent stats across some sessions, not sub-agents): total tokens cache hit rate wall time peak context 283M 98% 3963m 466k 258M 97% 2724m 467k 98M 94% 1353m 393k 67M 97% 614m 434k 75M 98% 1447m 498k 53M 99% 191m 375k 6M 96% 139m 124k 7M 98% 86m 118k 11M 99% 61m 147k I could see 256k context being sufficient for all sorts of work, even if intermediate progress/plan tracking files and docs might have to be used along the way, in addition to whatever plan support the harness has (for example, if you document something that will be relevant for load testing you might need that in 10 turns but not during the ones before then).
  - bkaae 3mo ago
    Thanks for sharing - is this a normal feature request you are implementing in this example or is this a project from scratch? Trying to get an idea of how your workflow compares to mine.
    - KronisLV 3mo ago
      Existing project and as usual, a few issues mashed together in one mostly coherent plan. The shorter Kimi sessions I mentioned were singular features, there 256k would be wholly adequate. A greenfield project would probably allow at least 2x fewer tokens to be used in most of those long tasks, but I was mostly after consistency and bug fixes along the way as needed.
  - smotched 3mo ago
    Once you have the plan you don't need to keep the 2hrs of research in the context (which is most of it) you can drop that plan into a file and start fresh for implementation.
    - XenophileJKO 3mo ago
      That is true, but I feel like sometimes if the conversation contains useful rational it can help to keep it. I think sometimes it is a judgement call, I will sometimes compress the context first. If I feel like the model and I explored a lot of options I won't want to keep context as it might be confusing. I think the more you use it the better judge you are of whether you should purge, compress, or just keep the context before executing the plan.
      - dandaka 3mo ago
        Models work best when they have short instructions and no noise. "Conversation [may] contain" also means "conversation has a lot of noise". It degrades performance and increases cost. I use handoff skill to ask model to write a prompt for itself.
- giancarlostoro 3mo ago
  I had Claude build me a Python-inspired .NET language that treats .NET as a first class citizen, and breaks backwards compatibility where some Python nuances don't really apply to .NET for. I was able to get it to build a sample ASP .NET Web application that ran on Culebral code. Haven't gone back to it, have been using Claude Code on a private project I'm still architecting. https://github.com/Giancarlos/Culebral https://github.com/Giancarlos/Culebral
  - pseudosavant 3mo ago
    That is pretty interesting. It compiles to the CLR?
    - giancarlostoro 3mo ago
      Yeah, that was the goal. I've wanted something like Python for .NET as someone who does both, but backwards compatibility to Pythons ecosystem is something that will always make it harder than just having something Python-like as much as possible, but runs on .NET the other thing I wound up going for Rusts approach to type hints, method signatures should always have type hints, and when instantiating objects or types for the first time, the value should be inferred from there.
      - pseudosavant 3mo ago
        In practice, how have you used this? Are there scenarios where you need to be deploying .NET but want the ergonomics to be more like Python?
- sheeshkebab 3mo ago
  Try doing a refactoring of some sort or larger new feature using just an agent on a moderately sized codebase, 256k will be compacting every few minutes, and result will be unusable.
  - hedgehog 3mo ago
    I do essentially all of my work with auto compact set to 250k and it's fine. It may be due to the way the project tooling is set up and the use of sub agents?
- _zoltan_ 3mo ago
  I have a couple projects where the background research is easily over 500k without writing any code, after ultracode subagents synthesis.
  - giancarlostoro 3mo ago
    I think the distinction is when the model decides to read code top to bottom vs when the model chooses to parse code indirectly to save on tokens, there's also AST tooling to let the model see project structure.

cyanydeez 3mo ago

I'm always curious what you guys are working on; every git repo I've run a local model on and stick below <100k to increase speed seems effective enough to scope patches and changes.

jdoe1337halo 3mo ago
They are just talking to the model in CC, while staying in a single thread. Doubt they have any actual coding knowledge to compartmentalize different problems in the codebase.
- giancarlostoro 3mo ago
  Depends on the programming language I'm using for a given project, and the domain I'm working with. I've been coding as a hobbyist for nearly two decades now (since my teens), professionally for 9 years, and was a TA before that for roughly 3 years at one of the best colleges for this field in the state (at least back then it was) where I taught other students about programming, in some cases I was their primary learning resource. But yeah, I have no idea about anything about software because you made an assumption off very little to go by.
KronisLV 3mo ago
My current Claude Code session has been going on for like 35 hours and has used up around 400 million tokens, thankfully almost all of those being cached (95-98%) - pretty typical for long form agentic work. First you spend like 2-3 hours working on a plan, once you have that you just tell the model to go and implement it, do adversarial sub-agent review loops before each commit and also make sure that all tooling and tests pass (including coverage requirements). You do need to poke it in a slightly different direction every few hours, though. Not even any novel work, just some refactoring and SSE notification hardening, bug fixes, alongside environment tuning and getting rid of some bottlenecks (also migrated from Oracle to PostgreSQL but that's mostly done). That said, Kimi somehow manages to use less context in the main thread than Anthropic's models (even when you use sub-agents and also dynamic workflows in Claude Code), might have something to do with either how the model is tuned or their Kimi Code harness - because even in most of the longer form sessions it doesn't seem to fill up quite as quickly (note: because the kimi vis tool doesn't have a full summary view across all agents, these are the main long running agent stats across some sessions, not sub-agents): total tokens cache hit rate wall time peak context 283M 98% 3963m 466k 258M 97% 2724m 467k 98M 94% 1353m 393k 67M 97% 614m 434k 75M 98% 1447m 498k 53M 99% 191m 375k 6M 96% 139m 124k 7M 98% 86m 118k 11M 99% 61m 147k I could see 256k context being sufficient for all sorts of work, even if intermediate progress/plan tracking files and docs might have to be used along the way, in addition to whatever plan support the harness has (for example, if you document something that will be relevant for load testing you might need that in 10 turns but not during the ones before then).
- bkaae 3mo ago
  Thanks for sharing - is this a normal feature request you are implementing in this example or is this a project from scratch? Trying to get an idea of how your workflow compares to mine.
  - KronisLV 3mo ago
    Existing project and as usual, a few issues mashed together in one mostly coherent plan. The shorter Kimi sessions I mentioned were singular features, there 256k would be wholly adequate. A greenfield project would probably allow at least 2x fewer tokens to be used in most of those long tasks, but I was mostly after consistency and bug fixes along the way as needed.
- smotched 3mo ago
  Once you have the plan you don't need to keep the 2hrs of research in the context (which is most of it) you can drop that plan into a file and start fresh for implementation.
  - XenophileJKO 3mo ago
    That is true, but I feel like sometimes if the conversation contains useful rational it can help to keep it. I think sometimes it is a judgement call, I will sometimes compress the context first. If I feel like the model and I explored a lot of options I won't want to keep context as it might be confusing. I think the more you use it the better judge you are of whether you should purge, compress, or just keep the context before executing the plan.
    - dandaka 3mo ago
      Models work best when they have short instructions and no noise. "Conversation [may] contain" also means "conversation has a lot of noise". It degrades performance and increases cost. I use handoff skill to ask model to write a prompt for itself.
giancarlostoro 3mo ago
I had Claude build me a Python-inspired .NET language that treats .NET as a first class citizen, and breaks backwards compatibility where some Python nuances don't really apply to .NET for. I was able to get it to build a sample ASP .NET Web application that ran on Culebral code. Haven't gone back to it, have been using Claude Code on a private project I'm still architecting. https://github.com/Giancarlos/Culebral https://github.com/Giancarlos/Culebral
- pseudosavant 3mo ago
  That is pretty interesting. It compiles to the CLR?
  - giancarlostoro 3mo ago
    Yeah, that was the goal. I've wanted something like Python for .NET as someone who does both, but backwards compatibility to Pythons ecosystem is something that will always make it harder than just having something Python-like as much as possible, but runs on .NET the other thing I wound up going for Rusts approach to type hints, method signatures should always have type hints, and when instantiating objects or types for the first time, the value should be inferred from there.
    - pseudosavant 3mo ago
      In practice, how have you used this? Are there scenarios where you need to be deploying .NET but want the ergonomics to be more like Python?
sheeshkebab 3mo ago
Try doing a refactoring of some sort or larger new feature using just an agent on a moderately sized codebase, 256k will be compacting every few minutes, and result will be unusable.
- hedgehog 3mo ago
  I do essentially all of my work with auto compact set to 250k and it's fine. It may be due to the way the project tooling is set up and the use of sub agents?
_zoltan_ 3mo ago
I have a couple projects where the background research is easily over 500k without writing any code, after ultracode subagents synthesis.
- giancarlostoro 3mo ago
  I think the distinction is when the model decides to read code top to bottom vs when the model chooses to parse code indirectly to save on tokens, there's also AST tooling to let the model see project structure.