5 ms·
Have they documented the context window changes for Claude 4 anywhere? My (barely informed) understanding was one of the reasons Gemini 2.5 has been so useful i
by tptacek 1y ago
Have they documented the context window changes for Claude 4 anywhere? My (barely informed) understanding was one of the reasons Gemini 2.5 has been so useful is that it can handle huge amounts of context --- 50-70kloc?
- jbellis 1y agonot sure wym, it's in the headline of the article that Opus 4 has 200k context (same as sonnet 3.7 with the beta header)
- tptacek 1y agoWe might be looking at different articles? The string "200" appears nowhere in this one --- or I'm just wrong! But thanks!
- jbellis 1y agoMy mistake, I was in fact looking at one of the linked details pages
- esafak 1y agoThere's the nominal context length, and the effective one. You need a benchmark like the needle-in-a-haystack or RULER to determine the latter. https://github.com/NVIDIA/RULER https://github.com/NVIDIA/RULER
- minimaxir 1y agoContext window is unchanged for Sonnet. (200k in/64k out): https://docs.anthropic.com/en/docs/about-claude/models/overview https://docs.anthropic.com/en/docs/about-claude/models/overv... In practice, the 1M context of Gemini 2.5 isn't that much of a differentiator because larger context has diminishing returns on adherence to later tokens.
- Workaccount2 1y agoGemini's real strength is that it can stay on the ball even at 100k tokens in context.
- zamadatix 1y agoThe amount of degradation at a given context length isn't constant though so a model with 5x the context can either be completely useless or still better depending on the strength of the models you're comparing. Gemini actually does really great in both regards (context length and quality at length) but I'm not sure what a hard numbers comparison to the latest Claude models would look like. A good deep dive on the context scaling topic in general https://youtu.be/NHMJ9mqKeMQ https://youtu.be/NHMJ9mqKeMQ
- ashirviskas 1y agoIt's closer to <30k before performance degrades too much for 3.5/3.7. 200k/64k is meaningless in this context.
- jerjerjer 1y agoIs there a benchmark to measure real effective context length? Sure, gpt-4o has a context window of 128k, but it loses a lot from the beginning/middle.
- bigmadshoe 1y agoThey often publish "needle in a haystack" benchmarks that look very good, but my subjective experience with a large context is always bad. Maybe we need better benchmarks.
- brookst 1y agoHere's an older study that includes Claude 3.5: https://www.databricks.com/blog/long-context-rag-capabilities-openai-o1-and-google-gemini https://www.databricks.com/blog/long-context-rag-capabilitie...?
- evertedsphere 1y ago
- fblp 1y agoI wish they would increase the context window or better handle it when the prompt gets too long. Currently users get "prompt is too long" warnings suddenly which makes it a frustrating model to work with for long conversations, writing etc. Other tools may drop some prior context, or use RAG to help but they don't force you to start a new chat without warning.
- keeganpoppen 1y agocontext window size is super fake. if you don't have the right context, you don't get good output.