3 ms·
Maybe this is OT but I wonder if openai/anthropic have private versions of their models with wider context windows (4M tokens? 10M tokens?). We know that the u
by znpy 7d ago
Maybe this is OT but I wonder if openai/anthropic have private versions of their models with wider context windows (4M tokens? 10M tokens?).
We know that the us government usually has private/custom versions of technology available to the general public, but much better.
- ImaCake 7d agoMy vague understanding of the context window limitation is that it is largely a constraint of the model architecture. So maybe they have special extra long ctx, but it might just be a hard limit of the model itself.
- irthomasthomas 7d agoI doub't it. The main issue is not cost, though they do get expensive as context grows, but intelligence. A frontier model like fable becomes as dumb as haiku after 200k tokens. They have been stuck at ~1M context/200k useful context for 18 months, now, with little sign of advancement. A model with a 10M context window that retains it's intelligence up to 2M tokens would be a big breakthrough.
- znpy 7d ago> A model with a 10M context window that retains it's intelligence up to 2M tokens would be a big breakthrough. so the true next frontier might not be just raw intelligence but rather larger context window?
- deleted 7d ago[deleted]
- DenisM 7d agoOr better context curation - less lossy compression saving back to context. Maybe even jettisoning part context into an external semantic store instead of conpression. Or placing less data into context to start with. Or a combination of all those things.
- gyyyu 7d ago[dead]