3 ms·
To be frank, the long-context techniques that you're describing are still extremely limited in the length of context to consider, only a million-token order, an
by inciampati 2y ago
To be frank, the long-context techniques that you're describing are still extremely limited in the length of context to consider, only a million-token order, and extremely expensive to apply.
- imtringued 2y agoAnd yet quadratic attention is still unavoidable. Anything that lets you have sub quadratic attention is going to have an accuracy Vs performance tradeoff.
- inciampati 2y agoNo, you can make multiple passes across the data set. There's no reason that you have to use quadratic attention. I point to the shape of our own brains, which are limited in size. Yet, through multiple passes over information and random access, which we guide, we're able to process very large systems.