4 ms·
This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k
by kelnos 4mo ago
This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see.
(And I rarely fill the context window that far anyway when working on a single task, or a series of tasks that are related enough to warrant the same context; more typical is anywhere between 200k and 600k or so.)
I'm not saying that no one ever has this experience, but it's odd to me that some people see it so often that it warrants giving it a name.
- asd88 4mo agoI’ve had similar experiences with Fable. 70%+ context used out of 1M, still sharp and no memory issues.
- cyanydeez 4mo agoAs the gamblers say at the poker table: If you can't figure out who the mark is when you site down...
- fullstackchris 4mo agoThats another problem of this post, the author mentions Claude but not explicitely what models... 100k tokens "by lunch" is also not my finding, the newer models will hit that already right in the initial exploratory phase
- arcanemachiner 4mo agoReally depends on the project.
- stavros 4mo agoI found "by lunch" odd too, but considering that Claude wrote the article, it's not going to know specifics.
- arcanemachiner 4mo agoOpus 4.6 was on drugs past 200k, I skipped 4.7, 4.8 did good up to ~350k, and Fable did great beyond 400k, in my limited testing. The quality does appear to be trending upwards.
- throwaway314155 4mo ago> Opus 4.6 was on drugs past 200k Which drugs?
- justinclift 4mo agoThe way it hallucinates stuff, it'd probably be something in the LSD family. ;)
- aeonik 4mo agoCombine it with meth and sleep deprivation and that could explain it.
- nijave 4mo agoShrooms, sometimes crack
- Bolwin 4mo agoI see this said often and find it insane given how many times I find opus models making basic recall mistakes at <100k tokens. Personally I consider < 60k to be the smart zone for opus. This is worse for opus 4.7 and 4.8 cause of the more granular tokenizer
- eterm 4mo ago60k is tiny, if it's making recall mistakes that early then you might have some false memories or incorrect instructions in your CLAUDE.md. 60k isn't much bigger than the system prompt.
- danielbln 4mo agoYeah 60k is ludicrous, I've barely seeded the context at that point and I don't see context related degradation until well into the 600-700k.
- embedding-shape 4mo ago> I've barely seeded the context at that point I think that's issue, rather than 60K being small. Most of the actual edits/changes I request to codex are solved within 100-150K tokens, beyond 200K I'd definitively try to restart the session as soon as I could as all models are horrible once you get across ~20% of the total context size. And this is while working on +million LOC codebases. Problem I guess is that there is no solid and concrete evidence of this (to me [and others seemingly] obvious) degradation, but should be easy to prove, yet no one has time to sit down and show it :) But the likelihood of a model getting minor details wrong once you're above some magical threshold between 15-20%, seems to skyrocket, and I hit that issue sufficient amount of times that now my workflow is trying to prevent that.
- qsera 4mo agoIn this thread: People tossing coins independently and fighting over the result they got.
- 4mo ago
- csomar 4mo agoI have a custom build command for a rust project (yarn build:lib) and my experience is 120k for GLM and roughly 200-300k for Opus. After that, they default to cargo build.
- trapexit 4mo agoMy projects have specific build/verify steps as well, and after a certain point Claude forgets to run them. I’m going to try a “No brown M&Ms” hook to halt Claude if it tries to run the default command instead of the instructed commands from CLAUDE.md. Perhaps this will be a good signal that a compacted or fresh session is needed at that point to avoid mistakes.
- csomar 4mo agoI mean, that’s basically the magic of the harness. The whole thing that skyrocketted the intelligence is that the harness (cli tool) prevent the LLM from editing the file before reading it. Can you imagine even a junior making such a mistake?
- tyleo 4mo agoI often push past 300k or so and I’ve absolutely worked at 800k but it’s an observable problem. Large context windows can work depending on the problem but I do feel more effective biasing towards small ones <300k.
- pdantix 4mo agoagreed. the claudes have been getting better and better with every release in this regard. opus 4.5 would start failing tool calls when approaching its 200k limit, opus 4.6 could get to ~300k before getting confused, opus 4.7 i could stretch to around 400k the dumb zone started, with opus 4.8 i've had sessions get over 500k comfortably. admittedly we only had limited time with fable, but i had a couple sessions get into 800-900k just fine.
- HarHarVeryFunny 4mo agoNot everybody is using the same model and harness as you, nor using the model the same way as you. Different models, and versions of models, use different types of attention, which affects their long-context performance, and no doubt also do different amounts/types of long context training. Different agents build context differently and implement context compaction differently. Unless someone else is using the same model as you, the same agent/harness as you, and doing very similar tasks, then there is no reason to suppose that their experience of model behavior relating to context size is going to be the same as yours.
- kelnos 4mo ago> then there is no reason to suppose that their experience of model behavior relating to context size is going to be the same as yours. Relax, I acknowledged this in my comment...
- saberience 4mo agoI never use Opus etc after 50% token usage (and from reading other devs blogs and on Twitter it seems this is a comment sentiment) because it falls off an intelligence cliff at that point. I mean, I really, really see intelligence tank at a certain amount of context usage. I always start a new session when any implementation work is starting or when starting a new plan. So I clean context before writing a plan, I clean context before any implementation of a plan. My first prompt is always putting enough of my own context, copy and pastes of docs, etc, to ensure the plan creation is good. Once the plan is made I clean the context and get Opus to implement said plan. Out of all the methodologies I've tried, this seems to be the best in terms of output quality.