3 ms·
That's a pretty weak article from Google, No real substance, Don't really explain or even hint at how they achieved such a long context window. It's just kind o
by ravetcofx 3y ago
That's a pretty weak article from Google, No real substance, Don't really explain or even hint at how they achieved such a long context window. It's just kind of a self-congratulatory pat on the back of an article :/
- kccqzy 3y agoThe article doesn't even profess to answer the "how" only the "what" as evidenced by this quote: > we asked the Google DeepMind project team to explain what long context windows are, and how this breakthrough experimental feature can help developers in many ways. At this time we can only speculate on how they achieved it. The best guess so far is RingAttention. See https://github.com/lucidrains/ring-attention-pytorch https://github.com/lucidrains/ring-attention-pytorch
- sandyarmstrong 3y ago> The article doesn't even profess to answer the "how" only the "what" Even the "what" isn't explained well enough if you're not already familiar with the topic. I'm not really sure who is meant to be the audience of this article. Big congrats to the team though, great competitive leap for them.