4 ms·

The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needless

The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is. You need to cut your prompt up. If you believe LLMs work, have the LLM help you shape the overall plan, and then have multiple sessions run each step in the plan without being bloated with the context of previous successful steps. I don't see LLMs being production-ready until the context rot and sampling problem is fixed forever. This has not occurred, and the big inference providers aren't even bothering to integrate any of the research on that subject. If anything, many of the bigger companies are actively making inference quality worse just to extend their runway a tiny bit farther before they go bankrupt. The only thing the article gets right is this: if you're serious about LLMs, abandon Big AI and infer locally only. This is the only way you have control over the quality of the output.

andai 18d ago
The Claude Code system prompt was >50KB, though I think they trimmed it down heavily recently. (The newer models don't need as much hand-holding.)
- embedding-shape 18d ago
  Yeah, strong evidence for what parent says is correct. Been my experience as well, especially with local (smaller) models but also SOTA. The less instructions you have, the better they get at following them. Conflicting instructions is like poison, and it's harder to find those conflicting parts the longer the prompt is too.
  - cyanydeez 18d ago
    the longer the context grows, the greater the probability that it generates ambiguity, and the probability that it makes mistakes approaches 1. A system prompt should be looked at like a starting point and a direction, but not a giant atlas map of everything it may be asked to do. I've been playing around with Qwen3.8-Next-Flash that has great logic, recall, roleplaying, etc. From what I can tell, it's definitely on part with the SOTA a few months ago, and my opinion, it likely is around the pinnacle of advancement without high inefficiency in preparing with the current LLM recipes. Further, I'm of the opinion America's SOTA is hitting the real cost-sigmoid and there's no singularity in site. These things will hack the planet if you put them in a group of agents and tell them to do it; but as context grows, the probability that they can answer "how many r's are in strawberry" goes down. No amount of parameters it going to erradicate that. But I digress, my new stage of working with LLMs is figuring out how to use Qwen3.6-35B-A3B as the entry point to collect the context, and then use the big boys Qwen3.8 to make the edits, then degrade back and forth to complete changes. There's no harness yet for this, but there's clearly an intelligent way to setup a engineering harness. And I do understand people have codebases that simply can't easily live in the smaller (~128k-256k) context windows, but instead of porting your codebase to another novel language, breaking it into context-aware components would make it closer to what these things can do. And I'll repeat: I don't think we're approaching the singularity of self-recursion primarily because the LLMs will duplicate errors, context poison, and whatever else they encounter and there's no human who can sit around correcting it constantly. The American AI apparatus should cut their models like the Chinese down and work on real problems and stop imbibing the singularity-watts-are-all-we-need drugs.
    - hedgehog 18d ago
      3.8 Next Flash seems like it is the first model usable unattended coding on a single desktop PC. The context window issue I think is a non-issue, at least to the extent that I already run Opus and Fable with auto-compact at 200k tokens. Even though theoretically those models support longer context I've never seen them perform as well at long lengths and the cached read cost starts to get really out of hand.
      - cyanydeez 18d ago
birdsongs 18d ago
Tbf, didn't read the article because it isn't applicable to me. I don't use system prompts or memory, I just use models stock and write the problem out. Is it really 250k? I had a long running autonomous Astra session today that got to about 600k and it finished fine with everything I asked it to do solved nicely. Opus 5 last week got to around 700k before I compacted between prompts, but also gave good performance. How do you all keep your context so low? Complex tasks just balloon it in my experience.
- JamesSwift 18d ago
  > How do you all keep your context so low? Complex tasks just balloon it in my experience. By breaking the problem into discrete steps and aggressively restarting the prompt from the current state after completion of said steps
  - embedding-shape 18d ago
    AKA "divide and conquer", how we programmers been fighting ever bigger and more complex problems since probably forever.
- gpugreg 18d ago
  The question isn't how you keep your context small, but how did your context get so big? A few common sources of bloat are long system prompts, unnecessary tools, unclear prompts, and scrawling code bases. To reduce system prompt and tool bloat, use a minimal harness (I wrote my own, but I've read that pi.dev is okay, too). To make your prompts more precise, tell the LLM which files it has to read (or at least where it should start), so it does not have to search as much. This also reduces the change of misunderstandings and makes the LLM adhere to existing practices. To keep your code base in check, tell the LLM (in a new session) to review the code and refactor from time to time. When a task is done, start a new session. If you find that you have to repeat a lot of information in your next prompt, put the information in a file so you can reference it in the future (aka documentation).
  - birdsongs 18d ago
gchamonlive 18d ago
> your prompt is confusing, unfocused, and doesn't work right on any LLM You are assuming the entirety of the prompt is human prose, but it could be sets of data so the agent doesn't have to collect it every time, like program interfaces, commands, views, databases, tables, data models etc... I could see this scale to multiple kiltobytes of metadata in the prompt easily.
- DiabloD3 18d ago
  That usually ends up being a poor use of LLMs, and is an unsolved problem with LLMs. RAG was supposed to be the way out on that, and ended up being mostly abandoned.
  - gchamonlive 18d ago
    That doesn't make much sense to me because this is in nature much like how harnesses operate: launch a bunch of exploratory subagents to search and retrieve evidence to use in the actual prompt. Think of it as caching this end result so you don't have to re-fetch in the codebase.
    - DiabloD3 18d ago
      That's the other way of doing it, which solves the context rot problem in a more complex way. The model at the top says, "hey, sub-agent, go figure out the answer to this question and give me the answer", and that sub-agent can go consume 250k+ context to return an answer that might be a couple of words, and thus not contaminate the main context with that now thrown-away context. However, this is not something that is inherently part of models or inference engine, but part of the harness. Harnesses are very hit and miss, and are not integrated into the stack, and I think that will have to happen eventually. Like, conceptually similar to an LLM performing a tool call that just calls itself recursively, I think this would go a long way to making LLMs more viable for being an actual product people could conceivably want.
      - docjay 18d ago
hermitShell 18d ago
> "until the context rot and sampling problem is fixed forever" I agree, prompt adherence seems to get worse when operating on large inputs. Does anyone have some notion of the SOTA with this? Can we expect big improvements by this time next year? (hopefully in open weights)
- DiabloD3 18d ago
  A lot of this is managed by the inference engine, and has nothing to do with the model. Models that use, for example, sparse attention mechanisms are just trying to make the bad situation slightly less bad, such as using less RAM for context (thus requiring less context quantization) or using less bandwidth (thus running faster). If people keep using temp, top-k, top-p, and min-p, and nothing else for samplers, we're ignoring ~3 years of sampling research that virtually eliminates the worst of context rot issues.
fennecbutt 18d ago
Attention is all you need. And there's only so much attention to go around.
lowbloodsugar 18d ago
> you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is. This was certainly true when I first tried the new models with a 1M context. After 200k things got weird pretty fast. I haven’t had that problem since Opus 4.8. I’m regularly bumping against 800k tokens in “lazy” adhoc sessions. “Lazy” in that I ought to do as you suggest, in the way that I ought to refactor this code, I ought to factor out the meat of this session, but in the moment it’s still producing useful output! Tool harness is a force multiplier too: tools that put all tool use in subagents are incredibly frugal with the main chat session.
chrisweekly 18d ago
The "dumb zone" threshold is fuzzy but comes waay before 250k tokens. Like half that.
varsha_saini 18d ago
[flagged]
sbnmkatoch 18d ago
[flagged]

andai 18d ago

The Claude Code system prompt was >50KB, though I think they trimmed it down heavily recently. (The newer models don't need as much hand-holding.)

embedding-shape 18d ago
Yeah, strong evidence for what parent says is correct. Been my experience as well, especially with local (smaller) models but also SOTA. The less instructions you have, the better they get at following them. Conflicting instructions is like poison, and it's harder to find those conflicting parts the longer the prompt is too.
- cyanydeez 18d ago
  the longer the context grows, the greater the probability that it generates ambiguity, and the probability that it makes mistakes approaches 1. A system prompt should be looked at like a starting point and a direction, but not a giant atlas map of everything it may be asked to do. I've been playing around with Qwen3.8-Next-Flash that has great logic, recall, roleplaying, etc. From what I can tell, it's definitely on part with the SOTA a few months ago, and my opinion, it likely is around the pinnacle of advancement without high inefficiency in preparing with the current LLM recipes. Further, I'm of the opinion America's SOTA is hitting the real cost-sigmoid and there's no singularity in site. These things will hack the planet if you put them in a group of agents and tell them to do it; but as context grows, the probability that they can answer "how many r's are in strawberry" goes down. No amount of parameters it going to erradicate that. But I digress, my new stage of working with LLMs is figuring out how to use Qwen3.6-35B-A3B as the entry point to collect the context, and then use the big boys Qwen3.8 to make the edits, then degrade back and forth to complete changes. There's no harness yet for this, but there's clearly an intelligent way to setup a engineering harness. And I do understand people have codebases that simply can't easily live in the smaller (~128k-256k) context windows, but instead of porting your codebase to another novel language, breaking it into context-aware components would make it closer to what these things can do. And I'll repeat: I don't think we're approaching the singularity of self-recursion primarily because the LLMs will duplicate errors, context poison, and whatever else they encounter and there's no human who can sit around correcting it constantly. The American AI apparatus should cut their models like the Chinese down and work on real problems and stop imbibing the singularity-watts-are-all-we-need drugs.
  - hedgehog 18d ago
    3.8 Next Flash seems like it is the first model usable unattended coding on a single desktop PC. The context window issue I think is a non-issue, at least to the extent that I already run Opus and Fable with auto-compact at 200k tokens. Even though theoretically those models support longer context I've never seen them perform as well at long lengths and the cached read cost starts to get really out of hand.
    - cyanydeez 18d ago

birdsongs 18d ago

Tbf, didn't read the article because it isn't applicable to me. I don't use system prompts or memory, I just use models stock and write the problem out. Is it really 250k? I had a long running autonomous Astra session today that got to about 600k and it finished fine with everything I asked it to do solved nicely. Opus 5 last week got to around 700k before I compacted between prompts, but also gave good performance. How do you all keep your context so low? Complex tasks just balloon it in my experience.

JamesSwift 18d ago
> How do you all keep your context so low? Complex tasks just balloon it in my experience. By breaking the problem into discrete steps and aggressively restarting the prompt from the current state after completion of said steps
- embedding-shape 18d ago
  AKA "divide and conquer", how we programmers been fighting ever bigger and more complex problems since probably forever.
gpugreg 18d ago
The question isn't how you keep your context small, but how did your context get so big? A few common sources of bloat are long system prompts, unnecessary tools, unclear prompts, and scrawling code bases. To reduce system prompt and tool bloat, use a minimal harness (I wrote my own, but I've read that pi.dev is okay, too). To make your prompts more precise, tell the LLM which files it has to read (or at least where it should start), so it does not have to search as much. This also reduces the change of misunderstandings and makes the LLM adhere to existing practices. To keep your code base in check, tell the LLM (in a new session) to review the code and refactor from time to time. When a task is done, start a new session. If you find that you have to repeat a lot of information in your next prompt, put the information in a file so you can reference it in the future (aka documentation).
- birdsongs 18d ago

gchamonlive 18d ago

> your prompt is confusing, unfocused, and doesn't work right on any LLM You are assuming the entirety of the prompt is human prose, but it could be sets of data so the agent doesn't have to collect it every time, like program interfaces, commands, views, databases, tables, data models etc... I could see this scale to multiple kiltobytes of metadata in the prompt easily.

DiabloD3 18d ago
That usually ends up being a poor use of LLMs, and is an unsolved problem with LLMs. RAG was supposed to be the way out on that, and ended up being mostly abandoned.
- gchamonlive 18d ago
  That doesn't make much sense to me because this is in nature much like how harnesses operate: launch a bunch of exploratory subagents to search and retrieve evidence to use in the actual prompt. Think of it as caching this end result so you don't have to re-fetch in the codebase.
  - DiabloD3 18d ago
    That's the other way of doing it, which solves the context rot problem in a more complex way. The model at the top says, "hey, sub-agent, go figure out the answer to this question and give me the answer", and that sub-agent can go consume 250k+ context to return an answer that might be a couple of words, and thus not contaminate the main context with that now thrown-away context. However, this is not something that is inherently part of models or inference engine, but part of the harness. Harnesses are very hit and miss, and are not integrated into the stack, and I think that will have to happen eventually. Like, conceptually similar to an LLM performing a tool call that just calls itself recursively, I think this would go a long way to making LLMs more viable for being an actual product people could conceivably want.
    - docjay 18d ago

hermitShell 18d ago

> "until the context rot and sampling problem is fixed forever" I agree, prompt adherence seems to get worse when operating on large inputs. Does anyone have some notion of the SOTA with this? Can we expect big improvements by this time next year? (hopefully in open weights)

DiabloD3 18d ago
A lot of this is managed by the inference engine, and has nothing to do with the model. Models that use, for example, sparse attention mechanisms are just trying to make the bad situation slightly less bad, such as using less RAM for context (thus requiring less context quantization) or using less bandwidth (thus running faster). If people keep using temp, top-k, top-p, and min-p, and nothing else for samplers, we're ignoring ~3 years of sampling research that virtually eliminates the worst of context rot issues.

fennecbutt 18d ago

Attention is all you need. And there's only so much attention to go around.

lowbloodsugar 18d ago

> you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is. This was certainly true when I first tried the new models with a 1M context. After 200k things got weird pretty fast. I haven’t had that problem since Opus 4.8. I’m regularly bumping against 800k tokens in “lazy” adhoc sessions. “Lazy” in that I ought to do as you suggest, in the way that I ought to refactor this code, I ought to factor out the meat of this session, but in the moment it’s still producing useful output! Tool harness is a force multiplier too: tools that put all tool use in subagents are incredibly frugal with the main chat session.

chrisweekly 18d ago

The "dumb zone" threshold is fuzzy but comes waay before 250k tokens. Like half that.

varsha_saini 18d ago

[flagged]

sbnmkatoch 18d ago

[flagged]