3 ms·

Context window is unchanged for Sonnet. (200k in/64k out): https://docs.anthropic.com/en/docs/about-claude/models/overview https://docs.anthropic.com/en/docs/ab

Context window is unchanged for Sonnet. (200k in/64k out): https://docs.anthropic.com/en/docs/about-claude/models/overview https://docs.anthropic.com/en/docs/about-claude/models/overv... In practice, the 1M context of Gemini 2.5 isn't that much of a differentiator because larger context has diminishing returns on adherence to later tokens.

Workaccount2 1y ago
Gemini's real strength is that it can stay on the ball even at 100k tokens in context.
zamadatix 1y ago
The amount of degradation at a given context length isn't constant though so a model with 5x the context can either be completely useless or still better depending on the strength of the models you're comparing. Gemini actually does really great in both regards (context length and quality at length) but I'm not sure what a hard numbers comparison to the latest Claude models would look like. A good deep dive on the context scaling topic in general https://youtu.be/NHMJ9mqKeMQ https://youtu.be/NHMJ9mqKeMQ
ashirviskas 1y ago
It's closer to <30k before performance degrades too much for 3.5/3.7. 200k/64k is meaningless in this context.
- jerjerjer 1y ago
  Is there a benchmark to measure real effective context length? Sure, gpt-4o has a context window of 128k, but it loses a lot from the beginning/middle.
  - bigmadshoe 1y ago
    They often publish "needle in a haystack" benchmarks that look very good, but my subjective experience with a large context is always bad. Maybe we need better benchmarks.
  - brookst 1y ago
    Here's an older study that includes Claude 3.5: https://www.databricks.com/blog/long-context-rag-capabilities-openai-o1-and-google-gemini https://www.databricks.com/blog/long-context-rag-capabilitie...?
  - evertedsphere 1y ago
    ruler https://arxiv.org/abs/2404.06654 https://arxiv.org/abs/2404.06654 nolima https://arxiv.org/abs/2502.05167 https://arxiv.org/abs/2502.05167
Rudybega 1y ago
I'm going to have to heavily disagree. Gemini 2.5 Pro has super impressive performance on large context problems. I routinely drive it up to 4-500k tokens in my coding agent. It's the only model where that much context produces even remotely useful results. I think it also crushes most of the benchmarks for long context performance. I believe on MRCR (multi round coreference resolution) it beats pretty much any other model's performance at 128k at 1M tokens (o3 may have changed this).
- egamirorrim 1y ago
  OOI what coding agent are you managing to get to work nicely with G2.5 Pro?
  - Rudybega 1y ago
    I mostly use Roo Code inside visual studio. The modes are awesome for managing context length within a discrete unit of work.
- alasano 1y ago
  I find that it consistently breaks around that exact range you specified. In the sense that reliability falls off a cliff, even though I've used it successfully close to the 1M token limit. At 500k+ I will define a task and it will suddenly panic and go back to a previous task that we just fully completed.
- l2silver 1y ago
  Is that a codebase you're giving it?
- vharish 1y ago
  Totally agreed on this. The context size is what made me switch to Gemini. Compared to Gemini, Claude's context window length is a joke. Particularly for indie projects, you can essentially dump the entire code into it and with pro reasoning model, it's all handled pretty well.
  - 1y ago
michaelbrave 1y ago
I've had a lot of fun using Gemini's large context. I scrape a reddit discussion with 7k responses, and have gemini synthesize it and categorize it, and by the time it's done and I have a few back and fourths with it I've gotten half of a book written. That said I have noticed that if I try to give it additional threads to compare and contrast once it hits around the 300-500k tokens it starts to hallucinate more and forget things more.
VeejayRampay 1y ago
that is just not correct, it's a big differentiator
strangescript 1y ago
Yeah, but why aren't they attacking that problem? Is it just impossible, because it would be a really simple win with regards to coding. I am huge enthusiast, but I am starting to feel a peak.

Workaccount2 1y ago

Gemini's real strength is that it can stay on the ball even at 100k tokens in context.

zamadatix 1y ago

The amount of degradation at a given context length isn't constant though so a model with 5x the context can either be completely useless or still better depending on the strength of the models you're comparing. Gemini actually does really great in both regards (context length and quality at length) but I'm not sure what a hard numbers comparison to the latest Claude models would look like. A good deep dive on the context scaling topic in general https://youtu.be/NHMJ9mqKeMQ https://youtu.be/NHMJ9mqKeMQ

ashirviskas 1y ago

It's closer to <30k before performance degrades too much for 3.5/3.7. 200k/64k is meaningless in this context.

jerjerjer 1y ago
Is there a benchmark to measure real effective context length? Sure, gpt-4o has a context window of 128k, but it loses a lot from the beginning/middle.
- bigmadshoe 1y ago
  They often publish "needle in a haystack" benchmarks that look very good, but my subjective experience with a large context is always bad. Maybe we need better benchmarks.
- brookst 1y ago
  Here's an older study that includes Claude 3.5: https://www.databricks.com/blog/long-context-rag-capabilities-openai-o1-and-google-gemini https://www.databricks.com/blog/long-context-rag-capabilitie...?
- evertedsphere 1y ago
  ruler https://arxiv.org/abs/2404.06654 https://arxiv.org/abs/2404.06654 nolima https://arxiv.org/abs/2502.05167 https://arxiv.org/abs/2502.05167

Rudybega 1y ago

I'm going to have to heavily disagree. Gemini 2.5 Pro has super impressive performance on large context problems. I routinely drive it up to 4-500k tokens in my coding agent. It's the only model where that much context produces even remotely useful results. I think it also crushes most of the benchmarks for long context performance. I believe on MRCR (multi round coreference resolution) it beats pretty much any other model's performance at 128k at 1M tokens (o3 may have changed this).

egamirorrim 1y ago
OOI what coding agent are you managing to get to work nicely with G2.5 Pro?
- Rudybega 1y ago
  I mostly use Roo Code inside visual studio. The modes are awesome for managing context length within a discrete unit of work.
alasano 1y ago
I find that it consistently breaks around that exact range you specified. In the sense that reliability falls off a cliff, even though I've used it successfully close to the 1M token limit. At 500k+ I will define a task and it will suddenly panic and go back to a previous task that we just fully completed.
l2silver 1y ago
Is that a codebase you're giving it?
vharish 1y ago
Totally agreed on this. The context size is what made me switch to Gemini. Compared to Gemini, Claude's context window length is a joke. Particularly for indie projects, you can essentially dump the entire code into it and with pro reasoning model, it's all handled pretty well.
- 1y ago

michaelbrave 1y ago

I've had a lot of fun using Gemini's large context. I scrape a reddit discussion with 7k responses, and have gemini synthesize it and categorize it, and by the time it's done and I have a few back and fourths with it I've gotten half of a book written. That said I have noticed that if I try to give it additional threads to compare and contrast once it hits around the 300-500k tokens it starts to hallucinate more and forget things more.

VeejayRampay 1y ago

that is just not correct, it's a big differentiator

strangescript 1y ago

Yeah, but why aren't they attacking that problem? Is it just impossible, because it would be a really simple win with regards to coding. I am huge enthusiast, but I am starting to feel a peak.