3 ms·
It's almost certainly not quadratic at 1M. This would be wildly infeasible at scale. 10^6^2 = 10^12. That's a trillion things. They are probably doing somethin
by bob1029 8mo ago
It's almost certainly not quadratic at 1M. This would be wildly infeasible at scale. 10^6^2 = 10^12. That's a trillion things.
They are probably doing something like putting the original user prompt into the model's environment and providing special tools to the model, along with iterative execution, to fully process the entire context over multiple invokes.
I think the Recursive Language Model paper has a very good take on how this might go. I've seen really good outcomes in my local experimentation around this concept:
https://arxiv.org/abs/2512.24601 https://arxiv.org/abs/2512.24601
You can get exponential scaling with proper symbolic stack frames. Handling a gigabyte of context is feasible, assuming it fits the depth first search pattern.
- cubefox 8mo agoSo that would mean it's not "really" a 1M context window. I guess this is more plausible than a linear architecture like MAMBA or GDN.