3 ms·
I think Anthropic and OpenAI could also have offered a one million context window a while ago. The relevant architecture breakthrough was probably when a linear
by cubefox 3y ago
I think Anthropic and OpenAI could also have offered a one million context window a while ago. The relevant architecture breakthrough was probably when a linear increase in context length only required a linear increase in inference compute instead of a quadratic one. Anthropic and then OpenAI achieved linear context compute scaling before an architecture for it was published publicly (MAMBA paper).
- bearjaws 3y agoThe problem is, the 128k window performed terribly and showed that attention was mostly limited to the first and last 20%. Increasing it to 1M just means even more data is ignored.
- cubefox 3y agoMaybe their architecture wasn't as good as MAMBA and Google could use the better architecture thanks to being late to the game...