2 ms·

Tbf, didn't read the article because it isn't applicable to me. I don't use system prompts or memory, I just use models stock and write the problem out. Is it

Tbf, didn't read the article because it isn't applicable to me. I don't use system prompts or memory, I just use models stock and write the problem out. Is it really 250k? I had a long running autonomous Astra session today that got to about 600k and it finished fine with everything I asked it to do solved nicely. Opus 5 last week got to around 700k before I compacted between prompts, but also gave good performance. How do you all keep your context so low? Complex tasks just balloon it in my experience.

JamesSwift 20d ago
> How do you all keep your context so low? Complex tasks just balloon it in my experience. By breaking the problem into discrete steps and aggressively restarting the prompt from the current state after completion of said steps
- embedding-shape 20d ago
  AKA "divide and conquer", how we programmers been fighting ever bigger and more complex problems since probably forever.
gpugreg 20d ago
The question isn't how you keep your context small, but how did your context get so big? A few common sources of bloat are long system prompts, unnecessary tools, unclear prompts, and scrawling code bases. To reduce system prompt and tool bloat, use a minimal harness (I wrote my own, but I've read that pi.dev is okay, too). To make your prompts more precise, tell the LLM which files it has to read (or at least where it should start), so it does not have to search as much. This also reduces the change of misunderstandings and makes the LLM adhere to existing practices. To keep your code base in check, tell the LLM (in a new session) to review the code and refactor from time to time. When a task is done, start a new session. If you find that you have to repeat a lot of information in your next prompt, put the information in a file so you can reference it in the future (aka documentation).
- birdsongs 20d ago
  I think I'm just slightly higher level, it seems. I don't give it small tasks, I give it ticket level prompts and let it decide what to do. (It being frontier models). I don't have a harness or prompt. Just VS code integration through my company. All my prompts are from scratch and the only context it has. Lately I haven't been saying go to X file and change Y, I say you have SSH keys to the embedded hardware: plan, deploy, and test latency improvements to the X stage of the image processing pipeline, parity test output images to avoid regressions, and target a 5ms latency improvement, ideas to look at are M, N, and P image processing techniques, I think latency is in this domain is mostly caused by Y, but verify, and let me know when you have a plan ready to implement. Then it churns for 4 hours, it builds and runs it's own test harness and profiler, and autonomously works until it has a changeset ready to go. That's how I get to 600k, and it seems to work fine? It got us 4.8ms today, through SIMD improvements and using a technique I hadn't thought of in place of a more expensive operation.
  - vosper 19d ago
    Pretty much my experience too. Using Sol in VSCode I just give it the problem and let it go. I don’t even look at or think about context. Works well for me.
  - hypfer 19d ago
    While this can work, especially for disposable code, doing this for non-disposable code is going to set you up for a very very _very_ rough awakening, the moment the models fail or your access to them fails. Make sure to regularly validate that what you think you understand is still what you actually understand. One can only safely offload so much.
    - birdsongs 19d ago
      I mean, thanks, but the PR was 60 lines of code and I've been doing this for over a decade, so I know how to critically evaluate the output. I don't want to do this, for what it's worth. But my company has gone all in, I'm pretty sure our C suite has psychosis, and if I don't use it I'm seen as lesser. I'm not planning to do this much longer. If I can't find a real SW engineering gig again soon I'm switching careers (to what - idk yet). Capable models or not, I don't want to be a manager, I want to solve problems, and they took the best part of my job from me. I spent the time it was churning practicing drawing.
inanutshellus 20d ago
My guess is your sessions are interactive conversations rather than planning a course of action then, separately, executing each piece.

JamesSwift 20d ago

> How do you all keep your context so low? Complex tasks just balloon it in my experience. By breaking the problem into discrete steps and aggressively restarting the prompt from the current state after completion of said steps

embedding-shape 20d ago
AKA "divide and conquer", how we programmers been fighting ever bigger and more complex problems since probably forever.

gpugreg 20d ago

The question isn't how you keep your context small, but how did your context get so big? A few common sources of bloat are long system prompts, unnecessary tools, unclear prompts, and scrawling code bases. To reduce system prompt and tool bloat, use a minimal harness (I wrote my own, but I've read that pi.dev is okay, too). To make your prompts more precise, tell the LLM which files it has to read (or at least where it should start), so it does not have to search as much. This also reduces the change of misunderstandings and makes the LLM adhere to existing practices. To keep your code base in check, tell the LLM (in a new session) to review the code and refactor from time to time. When a task is done, start a new session. If you find that you have to repeat a lot of information in your next prompt, put the information in a file so you can reference it in the future (aka documentation).

birdsongs 20d ago
I think I'm just slightly higher level, it seems. I don't give it small tasks, I give it ticket level prompts and let it decide what to do. (It being frontier models). I don't have a harness or prompt. Just VS code integration through my company. All my prompts are from scratch and the only context it has. Lately I haven't been saying go to X file and change Y, I say you have SSH keys to the embedded hardware: plan, deploy, and test latency improvements to the X stage of the image processing pipeline, parity test output images to avoid regressions, and target a 5ms latency improvement, ideas to look at are M, N, and P image processing techniques, I think latency is in this domain is mostly caused by Y, but verify, and let me know when you have a plan ready to implement. Then it churns for 4 hours, it builds and runs it's own test harness and profiler, and autonomously works until it has a changeset ready to go. That's how I get to 600k, and it seems to work fine? It got us 4.8ms today, through SIMD improvements and using a technique I hadn't thought of in place of a more expensive operation.
- vosper 19d ago
  Pretty much my experience too. Using Sol in VSCode I just give it the problem and let it go. I don’t even look at or think about context. Works well for me.
- hypfer 19d ago
  While this can work, especially for disposable code, doing this for non-disposable code is going to set you up for a very very _very_ rough awakening, the moment the models fail or your access to them fails. Make sure to regularly validate that what you think you understand is still what you actually understand. One can only safely offload so much.
  - birdsongs 19d ago
    I mean, thanks, but the PR was 60 lines of code and I've been doing this for over a decade, so I know how to critically evaluate the output. I don't want to do this, for what it's worth. But my company has gone all in, I'm pretty sure our C suite has psychosis, and if I don't use it I'm seen as lesser. I'm not planning to do this much longer. If I can't find a real SW engineering gig again soon I'm switching careers (to what - idk yet). Capable models or not, I don't want to be a manager, I want to solve problems, and they took the best part of my job from me. I spent the time it was churning practicing drawing.

inanutshellus 20d ago

My guess is your sessions are interactive conversations rather than planning a course of action then, separately, executing each piece.