4 ms·
Regarding the 10M tokens context, RingAttention has been shown [0] recently (by researchers, not ML engineers in a FAANG) to be able to scale to comparable (1M)
by nestorD 3y ago
Regarding the 10M tokens context, RingAttention has been shown [0] recently (by researchers, not ML engineers in a FAANG) to be able to scale to comparable (1M) context sizes (it does take work and a lot of GPUs).
[0]: https://news.ycombinator.com/item?id=39367141 https://news.ycombinator.com/item?id=39367141
- jebarker 3y ago> researchers, not ML engineers in a FAANG Why did you point out this distinction?
- nestorD 3y agoIt means they have significantly less means (to get a lot of GPUs letting them scale up in context length) and are likely less well-versed in optimization (which also helps with scaling up)[0]. I believe those two things together are likely enough to explain the difference between a 1M context length and a 10M context length. [0]: Which is not looking down on that particular research team, the vast majority of people have less means and optimization know-how than Google.
- vineyardmike 3y agoProbably to indicate that its research and not productized?