5 ms·

This is a problem with long context models. To put it as simple and as bluntly as possible: just because they claim you can use 1M tokens in your context doesn'

This is a problem with long context models. To put it as simple and as bluntly as possible: just because they claim you can use 1M tokens in your context doesn't mean its true and you should do that. Due to extreme quantization of models and the context's KV cache, and also just really shitty samplers provided to the user (hell, most are just getting rid of sampler knobs altogether), this problem will absolutely continue. Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.

simpaticoder 2mo ago
>Want it to go away, almost like magic? Local inference. Ah yes, magic that costs the same as a new car.
- DanHulton 2mo ago
  Not necessarily! If you have a semi-recent MacBook with even 32GB, you can run 20GB models that are pretty damn smart, with room to spare for the rest of your toolchain. If you’re reasonably connected to the code you’re writing and prompting the AI at the level of the code, not the level of the feature, you can get some fantastic results. Sure, it’s not the completely automated dreamland that’s been sold, but it’s still a speed-up on par from going from assembler to a higher-level language, which is still immense. And for effectively free, if you have a machine that would otherwise have been considered “developer-grade” for a lot of tasks anyway.
  - hedgehog 2mo ago
    What models have you found to work for which tasks? I find the local models very useful but not in a way that's replaced cloud models (yet, I remain hopeful).
  - gwerbin 2mo ago
    I tried this on my work M4 with 48 GB of memory and it's not as useful as I had hoped. If you set a large context window you end up using all your memory. Prefill is slow. And the 10-20B models are somewhat poor at understanding intent and tend to think in circles, so if you set a small thinking budget to control context size the model ends up answering based on some half formed thoughts. Or something like batch document summarization or image classification or whatever yeah it's great. But for "agentic" and coding workflows I think you would need a completely different harness setup and prompting style compared to what we are used to from the big proprietary models. I certainly have not yet figured out what that is.
- ux266478 2mo ago
  A good setup will cost you the same as a decent house in a major metropolitan area. And then the price of a used car to upgrade your electrical to handle that kind of load, and get the cooling setup you need.
- DiabloD3 2mo ago
  Tell me where I can buy a car for $1k or $2k.
mhitza 2mo ago
One of my early experiments last year with open source models and context size was with GPT-OSS 20B (the mxfp4, the "smart" 4-bit quantization). Even though it boasted a 128k context size it was bad at recall around 32k characters (didn't bother to implement the tokenizer for counting). The recall text was a simple hash generated, filler text from a dictionary file and a request at the end to return only the hash from the beginning of the prompt. Past 32k characters the response contained hallucinations of characters or full hashes. Just having large context size doesn't paint a full picture of capabilities, prompt adherence and other quality metrics.
firasd 2mo ago
This is conspiratorial speculation though. They did test many open weights models which spanned the full spectrum of performance: Nemotron 3 Ultra second-worst, GLM 5.2 top five https://arxiv.org/html/2607.25398v1 https://arxiv.org/html/2607.25398v1
eurekin 2mo ago
The needle benchmarks show, that models extended context works for the part, that can be explained as: "I can access/adress that part of the input". I have no idea, why in that context, the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time, which has to put an upper bound of long context support in some way. There's just more things to lose focus to (or, mismanage the limited attention heads resources - per token)
- hungryhobbit 2mo ago
  There are no "attention heads" or fixed number of things a model can pay attention to ... or at least not exactly. After every prompt the model decides "I have a weight of 1 to distribute between every token in my context". If you have ten tokens, each gets a weight of 0.1 ... ... except it's not that simple, because the LLMs don't distribute that "attention budget" equally. If your prompt was "where is Paris", then any tokens in context it can associate with Paris will get a greater share. If the word Paris is in your context, it might get 0.3 or 0.4 weight, and close by tokens might get 0.2, while other (unrelated) tokens get 0.03 or something. Now, add lots of context, and you start to see the problem: more context = greater distribution of the attention budget. Even if the prompt is about Paris, and the Paris tokens get higher weights, they are only getting (say) 0.002 ... while unrelated tokens are getting 0.001. All LLM "answers" are just math, computation, based on the context and those weights. If it can't "focus it's attention" because it's distributed among too much context, it's far more likely to miss the relevant tokens (eg. the Paris ones) and give you an answer that ignores key parts of context. But again, there's no fixed number of things it can pay attention to: it's a gradual degradation of the chance of it seeing key info it should, based on the amount of context. EDIT: Ok, I guess there are an internal thing called "attention heads"; TIL. They work exactly as I described (essentially I described a single "head"). Still, what I just wrote remains true: whether you have 1 head or 96 or ... however many your particular model has ... the fundamental issue isn't the number of heads. It's not like the LLM can pay attention to exactly (say) 96 things at once: each one of those heads has all of the available context "competing" for attention as described, and then the heads themselves "compete" to produce the answer (although not through the same mechanism; it's more like the weighting between heads is baked-into the LLM). At the end of the day, the problem isn't "there aren't enough heads to pay attention" and the LLM hits a fixed limit. The problem is each head is trying to weight the relevant stuff, and the more stuff there is to distribute weights through, the less likely it will get it right (and then when all the heads are combined to produce the final answer, it misses key context).
_fat_santa 2mo ago
At my org we've been building AI agents and one internal rule we have is to use at most 50% of the models context window with the recommendation to not go over 25% for large context window models. Anytime I see a "1M Context Window", my brain always goes "Gotcha so a 250k usable window"
- tedggh 2mo ago
  Why 25% and not 12% or 40%? Is this an arbitrary vibestimate or you had some tests done that pointed you to 25%? I am genuinely interested in how others deal with context issues. Also I would think the usable context window is variable depending on the task, for example summarizing documents vs analyzing large, scattered and complex instructions.
  - Analemma_ 2mo ago
    In my own experience (both Claude and Gemini 1M), you start running into problems around 60-70%. A 25% ceiling sounds like total cargo culting to me.
    - irthomasthomas 2mo ago
      Benchmarks show a large drop in quality as context grows. Your opus will be acting like haiku above 300k tokens.
  - horsawlarway 2mo ago
    I'm not the above poster, but my experiences generally line up with theirs. The first 20% of context is usually very solid. Agent stays on task, is responsive and focused on the original goal. The last 20% of context is usually pretty bad. Agent tends to loop, repeat, drift away into sub-goals (or non-goals). The middle 60% is a bit of a grab-bag. Sometimes it's fine, sometimes it's not. It varies by task, prompt, and luck. The above 25% and 50% limits seem like reasonable and simple rules to keep agents in the productive zones of usage. It's generally not worth doing extensive testing to try to find your exact threshold unless you're running exactly the same prompt at scale, on similar data. If you are in that spot, I won't actually recommend langchain tooling (I've been somewhat frustrated with it) but I think the concepts are worth understanding. You can do something like https://docs.langchain.com/oss/python/langchain/test/evals https://docs.langchain.com/oss/python/langchain/test/evals. Trajectory evals in particular seem to suffer as context length grows (https://docs.langchain.com/oss/python/langchain/test/evals#trajectory-match-evaluator https://docs.langchain.com/oss/python/langchain/test/evals#t...). I've used this for testing an agent that handles provisioning and permissions for developers with SaaS tooling (ex - if a developer asks, go provision an account [saml/oauth/etc] with the 3rd party, configure access, and let them know how to use it). I'm about 6 months out of date at this point, so I won't speak for the latest models, but what was available early this year absolutely suffers from extended context lengths.
eshack94 2mo ago
I've noticed this trend of the sampler knobs being removed. Can you explain why this might be the case?
barumrho 2mo ago
Would you mind sharing some of your setup? Which model and which params do you tweak (e.g. temperature)?
- DiabloD3 2mo ago
  See the end of https://news.ycombinator.com/item?id=49100144 https://news.ycombinator.com/item?id=49100144
derefr 2mo ago
Long-context models do work as advertised... just not when combined with multi-turn conversation sessions. It's my understanding that, for all current models having long-context capability, the "parts" of the model that allow them to do long-context processing, are mostly-untouched descendants of academic model architectures; where these academic models were designed around a very constrained use-case assumption: that anyone using more than a "normal" amount of context would specifically be trying to apply a reasonably-sized prompt to an unreasonably-sized block of plain-old embedded data, as a single-shot conversation. (Consider a task like e.g. "summarize this entire book." Context = [prompt] + [text of book].) This is the use-case all these architectures were refined and benchmarked around. And even today, this is still basically the only way these models' long-context processing capabilities work. --- But I think it goes deeper than that. It's not just about how we've made long-context work so far. It's about what "attending to context" really means, and what that implies about what we'll ever expect a long-context model to be able to do. I would describe long-context models as seeming to have two distinct types of "attention state" used when attending to the layer-1 input-vector / context window: - They have some small amount (e.g. 8-32k) of regular "high-quality" attention state. This is likely inherited from the base model they are trained on top of. - They also have a large amount of "low-quality" attention state. This is what the long-context training process has created. The "high-quality attention state" isn't limited to referencing the first 8-32k of the context window, to be clear. The high-quality attention mechanism is just limited to "choosing" 8-32k of arbitrary data samples, each from its own arbitrary position in the context window, to form a bounded-sized vector upon which high-quality attention is then computed. (You could think of this as the high-quality attention mechanism computing attention over the entire context, and then doing top-K sampling, i.e. zeroing out all but the K largest-valued elements of the resulting latent vector, where K=8-32k. That's not what it's actually doing—that'd require paying the very quadratic scaling cost that prevents us from just increasing the short-context window size—but it's a useful naive model for what it's doing.) --- It's harder to directly explain what the low-quality attention mechanism does, so let me try an analogy. (While no one can yet give a definitive accounting of the abstract computational processes the matrix-multiply-ops against a given LLM's weights encode, we can still speak of possibility-spaces, or analogies via information/computability theory on what sorts of abstract operations the LLM "must" be performing to achieve the results it achieves. This is that kind of thing.) Think of an LLM's evolved context-window attendance mechanisms as being somewhat akin to the LLM being a regular CPU program, that has access to a 1. block of raw addressable memory (i.e. the context itself), and 2. a block of pointers into that memory (i.e. the learned first-layer attention over the context.) Our LLM program's logic, through mutations during training, gradually evolves into a form where it could be seen as implicitly declaring and using a growing set of variables: some bound to static addresses in the block of raw memory (i.e. static context-window positions), and others bound to static addresses in the block of pointers, where the raw-memory-address-value of the pointer (and thus what data from the context-window it will read as when "dereferenced") is the result of a dynamic calculation over one or more other variables (i.e. an arbitrary weighting of the layer-1 input vector, where the resulting vector element is used as a selector, in layer 2, among other data that got sampled into "passthrough" vector-elements back in layer 1.) Under this analogy, a model's "high-quality attention state" is the set of "named" pointer variables that it directly references in its logic. As should be obvious, there can only be so many of these; you really fundamentally can't scale them up, without literally having "more logic" that defines and makes use of them (i.e. scaling up the size-in-weights of the model's layers.) A model's "low-quality attention state", on the other hand, is the block of pointer memory itself, which can be as large as you like. Presuming the computational substrate permits the logic to compute on the addresses of memory within that pointer-memory block and then double-dereference them to get context data (and it does seem to!), your LLM's evolved logic can do "dynamic random-access reads" of arbitrary parts of the long context. But, of course, the model won't be doing anything special with that data. It can attend to it, but not in a way that's unique to that data. It pulled in such data through generic logic that doesn't "know what it's looking at", after all. --- IMHO, this analogy helps to make clear that while giving a model a raw "large-context capability" might help said model to attend to more data, it doesn't really help in making a model able to attend to a larger prompt. Just by information theory, an instruction-following capability must imply that something analogous to logic keyed on those instructions gets embedded into the weights; and that logic, in order to activate and work, presumably requires its own state — i.e. requires that individual values from the context are getting attended into additional top-level named variables. And while the short-context attendance mechanism does that, the long-context attendance mechanism inherently does not / cannot. And this means that there is likely a practical upper limit (in terms of practical model-weight size + latent-vector size + etc) to "how much prompt" a model can follow during a single inference step. Regardless of how much data the model can "see" on a given inference step, its active "program logic" is only defined in terms of so many concurrent stateful computations. --- Thus: trying to use a current long-context model's "low quality" attention to attend to a large prompt is just going to result in nonsense. None of these model families have ever been trained to follow a million sentences of rules simultaneously when answering a question. They're just going to attend with their inherited bounded "high quality" attention to your prompt as best as they're able, while forgetting literally everything about the prompt that doesn't fit in that bounded "high quality" window. And anything that allows current long-context models to appear to interact well in multi-turn conversations with memory + KV caching + etc layers in operation — while also making use of its "low quality" attention to attend to large data — is some kind of hack, and will break like a hack.
satvikpendem 2mo ago
How do they go away with local models? It's a bug of all LLMs not just cloud vs local. As mentioned in another comment, they did test local models here too and those failed as well.
- ux266478 2mo ago
  As parent implies, they're testing the wrong control mechanism. Why are you using policies instead of real controls over the weights and inference pipeline? Well the answer is that VC-backed companies decided AI is not a domain expert tool for highly competent technical users, it's a magic oracle for the lowest common denominator. So you don't get any of the actually useful controls, just context engineering like that's fucking sane at all. It's like trying to program by navigating a git history. Not writing any new code, you don't have the ability to do that. No, you exclusively have the ability to move around a git history and cherry pick things. It's insulting that they want to charge money for this shit.
  - giancarlostoro 2mo ago
    > just context engineering like that's fucking sane at all. Meanwhile every model is at a battle to figure out how to stop you from jailbreaking it, which I assume takes a toll on your ability to prompt. Heck, if you read r/Claude and related subs during the early release of Fable there were a lot of complaints that it would downgrade you heavily.
  - TiccyRobby 2mo ago
    And what is the difference between cloud modeld vs local inference regarding this? It is still unclear in this thread - asking out of curiosity.
    - ux266478 2mo ago
      The ultimate difference isn't necessarily down to remote vs local. You can run a "local" model on the cloud the same way you could on a local machine and have all the same control over it. You can modify the weights, inject behavior, stack on arbitrary LoRAs, use whatever runtime you want, etc. The problem arises in the case of anthropic, openai, etc. where you can't do those things. I think they have LoRA services that they'll charge you for, but that's about the extent of it. Think of it like having a compiler and its source code, and the operating system it runs on vs simply having access to invoking it via a web service.
Aurornis 2mo ago
> Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away. This is just not true. Any local LLM you can host on consumer-accessible hardware has all of these defects, too. Adjusting the knobs doesn’t solve everything. The closest you can get to frontier performance is Kimi K3, but you’re not hosting that unless your budget is on the order of a nice house in a good metro area. I like my local LLMs as much as the next person and my office is currently uncomfortably warm from the amount of compute happening, but I would never agree that local LLMs solve all of the common LLM defects. This is peak wishful thinking. In my experience, the local models and even the larger ones that we can’t run at home suffer more from long context degradation than the frontier models. You are exactly right that you need to manage context length, but even at fp16/bf16 the local models have a lower ceiling for usable context length in my experience.
- slopinthebag 2mo ago
  Do you work for a frontier lab by any chance?
  - Aurornis 2mo ago
    No.
- dingnuts 2mo ago
  [dead]
- nzeid 2mo ago
  This is a bit of a strawman. The parent comment wasn't suggesting that the inherent defects magically disappear nor was it suggesting that local inference today is sufficient for all tasks. At the heart of it, self-hosting liberates your use cases from all the horribly opaque configuration, shadow prompting, etc. And local models are only getting better and more diverse every month.
  - joshmoody24 2mo ago
iririririr 2mo ago
this is cope (i mean, it happens a lot, but is not the cause of what is seem on the paper). I noticed even with plenty of context, the model sometimes chose a path that "fools" the attention layers to bypass some rules. you can notice that the rules are not ignored, they are sidestepped. it's very predictable emergent behaviour after you see it happen. with more context, it actually gets worse to the point you will see some things obfuscated even.
eikenberry 2mo ago
It seems like you mean to use your own agent (where you control the prompts), not local inference. A custom agent works just as well with cloud based models.
matheusmoreira 2mo ago
Just need the hardware industry to catch up with consumer demand. 200kUSD+ on a computer capable of local inference is simply a non-starter.

simpaticoder 2mo ago

>Want it to go away, almost like magic? Local inference. Ah yes, magic that costs the same as a new car.

DanHulton 2mo ago
Not necessarily! If you have a semi-recent MacBook with even 32GB, you can run 20GB models that are pretty damn smart, with room to spare for the rest of your toolchain. If you’re reasonably connected to the code you’re writing and prompting the AI at the level of the code, not the level of the feature, you can get some fantastic results. Sure, it’s not the completely automated dreamland that’s been sold, but it’s still a speed-up on par from going from assembler to a higher-level language, which is still immense. And for effectively free, if you have a machine that would otherwise have been considered “developer-grade” for a lot of tasks anyway.
- hedgehog 2mo ago
  What models have you found to work for which tasks? I find the local models very useful but not in a way that's replaced cloud models (yet, I remain hopeful).
- gwerbin 2mo ago
  I tried this on my work M4 with 48 GB of memory and it's not as useful as I had hoped. If you set a large context window you end up using all your memory. Prefill is slow. And the 10-20B models are somewhat poor at understanding intent and tend to think in circles, so if you set a small thinking budget to control context size the model ends up answering based on some half formed thoughts. Or something like batch document summarization or image classification or whatever yeah it's great. But for "agentic" and coding workflows I think you would need a completely different harness setup and prompting style compared to what we are used to from the big proprietary models. I certainly have not yet figured out what that is.
ux266478 2mo ago
A good setup will cost you the same as a decent house in a major metropolitan area. And then the price of a used car to upgrade your electrical to handle that kind of load, and get the cooling setup you need.
DiabloD3 2mo ago
Tell me where I can buy a car for $1k or $2k.

mhitza 2mo ago

One of my early experiments last year with open source models and context size was with GPT-OSS 20B (the mxfp4, the "smart" 4-bit quantization). Even though it boasted a 128k context size it was bad at recall around 32k characters (didn't bother to implement the tokenizer for counting). The recall text was a simple hash generated, filler text from a dictionary file and a request at the end to return only the hash from the beginning of the prompt. Past 32k characters the response contained hallucinations of characters or full hashes. Just having large context size doesn't paint a full picture of capabilities, prompt adherence and other quality metrics.

firasd 2mo ago

This is conspiratorial speculation though. They did test many open weights models which spanned the full spectrum of performance: Nemotron 3 Ultra second-worst, GLM 5.2 top five https://arxiv.org/html/2607.25398v1 https://arxiv.org/html/2607.25398v1

eurekin 2mo ago

The needle benchmarks show, that models extended context works for the part, that can be explained as: "I can access/adress that part of the input". I have no idea, why in that context, the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time, which has to put an upper bound of long context support in some way. There's just more things to lose focus to (or, mismanage the limited attention heads resources - per token)

hungryhobbit 2mo ago
There are no "attention heads" or fixed number of things a model can pay attention to ... or at least not exactly. After every prompt the model decides "I have a weight of 1 to distribute between every token in my context". If you have ten tokens, each gets a weight of 0.1 ... ... except it's not that simple, because the LLMs don't distribute that "attention budget" equally. If your prompt was "where is Paris", then any tokens in context it can associate with Paris will get a greater share. If the word Paris is in your context, it might get 0.3 or 0.4 weight, and close by tokens might get 0.2, while other (unrelated) tokens get 0.03 or something. Now, add lots of context, and you start to see the problem: more context = greater distribution of the attention budget. Even if the prompt is about Paris, and the Paris tokens get higher weights, they are only getting (say) 0.002 ... while unrelated tokens are getting 0.001. All LLM "answers" are just math, computation, based on the context and those weights. If it can't "focus it's attention" because it's distributed among too much context, it's far more likely to miss the relevant tokens (eg. the Paris ones) and give you an answer that ignores key parts of context. But again, there's no fixed number of things it can pay attention to: it's a gradual degradation of the chance of it seeing key info it should, based on the amount of context. EDIT: Ok, I guess there are an internal thing called "attention heads"; TIL. They work exactly as I described (essentially I described a single "head"). Still, what I just wrote remains true: whether you have 1 head or 96 or ... however many your particular model has ... the fundamental issue isn't the number of heads. It's not like the LLM can pay attention to exactly (say) 96 things at once: each one of those heads has all of the available context "competing" for attention as described, and then the heads themselves "compete" to produce the answer (although not through the same mechanism; it's more like the weighting between heads is baked-into the LLM). At the end of the day, the problem isn't "there aren't enough heads to pay attention" and the LLM hits a fixed limit. The problem is each head is trying to weight the relevant stuff, and the more stuff there is to distribute weights through, the less likely it will get it right (and then when all the heads are combined to produce the final answer, it misses key context).

_fat_santa 2mo ago

At my org we've been building AI agents and one internal rule we have is to use at most 50% of the models context window with the recommendation to not go over 25% for large context window models. Anytime I see a "1M Context Window", my brain always goes "Gotcha so a 250k usable window"

tedggh 2mo ago
Why 25% and not 12% or 40%? Is this an arbitrary vibestimate or you had some tests done that pointed you to 25%? I am genuinely interested in how others deal with context issues. Also I would think the usable context window is variable depending on the task, for example summarizing documents vs analyzing large, scattered and complex instructions.
- Analemma_ 2mo ago
  In my own experience (both Claude and Gemini 1M), you start running into problems around 60-70%. A 25% ceiling sounds like total cargo culting to me.
  - irthomasthomas 2mo ago
    Benchmarks show a large drop in quality as context grows. Your opus will be acting like haiku above 300k tokens.
- horsawlarway 2mo ago
  I'm not the above poster, but my experiences generally line up with theirs. The first 20% of context is usually very solid. Agent stays on task, is responsive and focused on the original goal. The last 20% of context is usually pretty bad. Agent tends to loop, repeat, drift away into sub-goals (or non-goals). The middle 60% is a bit of a grab-bag. Sometimes it's fine, sometimes it's not. It varies by task, prompt, and luck. The above 25% and 50% limits seem like reasonable and simple rules to keep agents in the productive zones of usage. It's generally not worth doing extensive testing to try to find your exact threshold unless you're running exactly the same prompt at scale, on similar data. If you are in that spot, I won't actually recommend langchain tooling (I've been somewhat frustrated with it) but I think the concepts are worth understanding. You can do something like https://docs.langchain.com/oss/python/langchain/test/evals https://docs.langchain.com/oss/python/langchain/test/evals. Trajectory evals in particular seem to suffer as context length grows (https://docs.langchain.com/oss/python/langchain/test/evals#trajectory-match-evaluator https://docs.langchain.com/oss/python/langchain/test/evals#t...). I've used this for testing an agent that handles provisioning and permissions for developers with SaaS tooling (ex - if a developer asks, go provision an account [saml/oauth/etc] with the 3rd party, configure access, and let them know how to use it). I'm about 6 months out of date at this point, so I won't speak for the latest models, but what was available early this year absolutely suffers from extended context lengths.

eshack94 2mo ago

I've noticed this trend of the sampler knobs being removed. Can you explain why this might be the case?

barumrho 2mo ago

Would you mind sharing some of your setup? Which model and which params do you tweak (e.g. temperature)?

DiabloD3 2mo ago
See the end of https://news.ycombinator.com/item?id=49100144 https://news.ycombinator.com/item?id=49100144

derefr 2mo ago

Long-context models do work as advertised... just not when combined with multi-turn conversation sessions. It's my understanding that, for all current models having long-context capability, the "parts" of the model that allow them to do long-context processing, are mostly-untouched descendants of academic model architectures; where these academic models were designed around a very constrained use-case assumption: that anyone using more than a "normal" amount of context would specifically be trying to apply a reasonably-sized prompt to an unreasonably-sized block of plain-old embedded data, as a single-shot conversation. (Consider a task like e.g. "summarize this entire book." Context = [prompt] + [text of book].) This is the use-case all these architectures were refined and benchmarked around. And even today, this is still basically the only way these models' long-context processing capabilities work. --- But I think it goes deeper than that. It's not just about how we've made long-context work so far. It's about what "attending to context" really means, and what that implies about what we'll ever expect a long-context model to be able to do. I would describe long-context models as seeming to have two distinct types of "attention state" used when attending to the layer-1 input-vector / context window: - They have some small amount (e.g. 8-32k) of regular "high-quality" attention state. This is likely inherited from the base model they are trained on top of. - They also have a large amount of "low-quality" attention state. This is what the long-context training process has created. The "high-quality attention state" isn't limited to referencing the first 8-32k of the context window, to be clear. The high-quality attention mechanism is just limited to "choosing" 8-32k of arbitrary data samples, each from its own arbitrary position in the context window, to form a bounded-sized vector upon which high-quality attention is then computed. (You could think of this as the high-quality attention mechanism computing attention over the entire context, and then doing top-K sampling, i.e. zeroing out all but the K largest-valued elements of the resulting latent vector, where K=8-32k. That's not what it's actually doing—that'd require paying the very quadratic scaling cost that prevents us from just increasing the short-context window size—but it's a useful naive model for what it's doing.) --- It's harder to directly explain what the low-quality attention mechanism does, so let me try an analogy. (While no one can yet give a definitive accounting of the abstract computational processes the matrix-multiply-ops against a given LLM's weights encode, we can still speak of possibility-spaces, or analogies via information/computability theory on what sorts of abstract operations the LLM "must" be performing to achieve the results it achieves. This is that kind of thing.) Think of an LLM's evolved context-window attendance mechanisms as being somewhat akin to the LLM being a regular CPU program, that has access to a 1. block of raw addressable memory (i.e. the context itself), and 2. a block of pointers into that memory (i.e. the learned first-layer attention over the context.) Our LLM program's logic, through mutations during training, gradually evolves into a form where it could be seen as implicitly declaring and using a growing set of variables: some bound to static addresses in the block of raw memory (i.e. static context-window positions), and others bound to static addresses in the block of pointers, where the raw-memory-address-value of the pointer (and thus what data from the context-window it will read as when "dereferenced") is the result of a dynamic calculation over one or more other variables (i.e. an arbitrary weighting of the layer-1 input vector, where the resulting vector element is used as a selector, in layer 2, among other data that got sampled into "passthrough" vector-elements back in layer 1.) Under this analogy, a model's "high-quality attention state" is the set of "named" pointer variables that it directly references in its logic. As should be obvious, there can only be so many of these; you really fundamentally can't scale them up, without literally having "more logic" that defines and makes use of them (i.e. scaling up the size-in-weights of the model's layers.) A model's "low-quality attention state", on the other hand, is the block of pointer memory itself, which can be as large as you like. Presuming the computational substrate permits the logic to compute on the addresses of memory within that pointer-memory block and then double-dereference them to get context data (and it does seem to!), your LLM's evolved logic can do "dynamic random-access reads" of arbitrary parts of the long context. But, of course, the model won't be doing anything special with that data. It can attend to it, but not in a way that's unique to that data. It pulled in such data through generic logic that doesn't "know what it's looking at", after all. --- IMHO, this analogy helps to make clear that while giving a model a raw "large-context capability" might help said model to attend to more data, it doesn't really help in making a model able to attend to a larger prompt. Just by information theory, an instruction-following capability must imply that something analogous to logic keyed on those instructions gets embedded into the weights; and that logic, in order to activate and work, presumably requires its own state — i.e. requires that individual values from the context are getting attended into additional top-level named variables. And while the short-context attendance mechanism does that, the long-context attendance mechanism inherently does not / cannot. And this means that there is likely a practical upper limit (in terms of practical model-weight size + latent-vector size + etc) to "how much prompt" a model can follow during a single inference step. Regardless of how much data the model can "see" on a given inference step, its active "program logic" is only defined in terms of so many concurrent stateful computations. --- Thus: trying to use a current long-context model's "low quality" attention to attend to a large prompt is just going to result in nonsense. None of these model families have ever been trained to follow a million sentences of rules simultaneously when answering a question. They're just going to attend with their inherited bounded "high quality" attention to your prompt as best as they're able, while forgetting literally everything about the prompt that doesn't fit in that bounded "high quality" window. And anything that allows current long-context models to appear to interact well in multi-turn conversations with memory + KV caching + etc layers in operation — while also making use of its "low quality" attention to attend to large data — is some kind of hack, and will break like a hack.

satvikpendem 2mo ago

How do they go away with local models? It's a bug of all LLMs not just cloud vs local. As mentioned in another comment, they did test local models here too and those failed as well.

ux266478 2mo ago
As parent implies, they're testing the wrong control mechanism. Why are you using policies instead of real controls over the weights and inference pipeline? Well the answer is that VC-backed companies decided AI is not a domain expert tool for highly competent technical users, it's a magic oracle for the lowest common denominator. So you don't get any of the actually useful controls, just context engineering like that's fucking sane at all. It's like trying to program by navigating a git history. Not writing any new code, you don't have the ability to do that. No, you exclusively have the ability to move around a git history and cherry pick things. It's insulting that they want to charge money for this shit.
- giancarlostoro 2mo ago
  > just context engineering like that's fucking sane at all. Meanwhile every model is at a battle to figure out how to stop you from jailbreaking it, which I assume takes a toll on your ability to prompt. Heck, if you read r/Claude and related subs during the early release of Fable there were a lot of complaints that it would downgrade you heavily.
- TiccyRobby 2mo ago
  And what is the difference between cloud modeld vs local inference regarding this? It is still unclear in this thread - asking out of curiosity.
  - ux266478 2mo ago
    The ultimate difference isn't necessarily down to remote vs local. You can run a "local" model on the cloud the same way you could on a local machine and have all the same control over it. You can modify the weights, inject behavior, stack on arbitrary LoRAs, use whatever runtime you want, etc. The problem arises in the case of anthropic, openai, etc. where you can't do those things. I think they have LoRA services that they'll charge you for, but that's about the extent of it. Think of it like having a compiler and its source code, and the operating system it runs on vs simply having access to invoking it via a web service.

Aurornis 2mo ago

> Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away. This is just not true. Any local LLM you can host on consumer-accessible hardware has all of these defects, too. Adjusting the knobs doesn’t solve everything. The closest you can get to frontier performance is Kimi K3, but you’re not hosting that unless your budget is on the order of a nice house in a good metro area. I like my local LLMs as much as the next person and my office is currently uncomfortably warm from the amount of compute happening, but I would never agree that local LLMs solve all of the common LLM defects. This is peak wishful thinking. In my experience, the local models and even the larger ones that we can’t run at home suffer more from long context degradation than the frontier models. You are exactly right that you need to manage context length, but even at fp16/bf16 the local models have a lower ceiling for usable context length in my experience.

slopinthebag 2mo ago
Do you work for a frontier lab by any chance?
- Aurornis 2mo ago
  No.
dingnuts 2mo ago
[dead]
nzeid 2mo ago
This is a bit of a strawman. The parent comment wasn't suggesting that the inherent defects magically disappear nor was it suggesting that local inference today is sufficient for all tasks. At the heart of it, self-hosting liberates your use cases from all the horribly opaque configuration, shadow prompting, etc. And local models are only getting better and more diverse every month.
- joshmoody24 2mo ago

iririririr 2mo ago

this is cope (i mean, it happens a lot, but is not the cause of what is seem on the paper). I noticed even with plenty of context, the model sometimes chose a path that "fools" the attention layers to bypass some rules. you can notice that the rules are not ignored, they are sidestepped. it's very predictable emergent behaviour after you see it happen. with more context, it actually gets worse to the point you will see some things obfuscated even.

eikenberry 2mo ago

It seems like you mean to use your own agent (where you control the prompts), not local inference. A custom agent works just as well with cloud based models.

matheusmoreira 2mo ago

Just need the hardware industry to catch up with consumer demand. 200kUSD+ on a computer capable of local inference is simply a non-starter.