4 ms·
What is a long context window?
- marban 3y agotldr: 1M/128K = 7.8x longer
- sandyarmstrong 3y agoThe window was 32k before. 128k was their goal for the new version, and is the new standard window size. 1M is in private preview. 10M has been demonstrated internally.
- ArunRaja 3y ago1M/32k = 31x longer
- ravetcofx 3y agoThat's a pretty weak article from Google, No real substance, Don't really explain or even hint at how they achieved such a long context window. It's just kind of a self-congratulatory pat on the back of an article :/
- kccqzy 3y agoThe article doesn't even profess to answer the "how" only the "what" as evidenced by this quote: > we asked the Google DeepMind project team to explain what long context windows are, and how this breakthrough experimental feature can help developers in many ways. At this time we can only speculate on how they achieved it. The best guess so far is RingAttention. See https://github.com/lucidrains/ring-attention-pytorch https://github.com/lucidrains/ring-attention-pytorch
- sandyarmstrong 3y ago> The article doesn't even profess to answer the "how" only the "what" Even the "what" isn't explained well enough if you're not already familiar with the topic. I'm not really sure who is meant to be the audience of this article. Big congrats to the team though, great competitive leap for them.