Probably once context lengths get really long it might be better to use ANN rather than exact attention at inference time. I would imagine this would only pay o

Probably once context lengths get really long it might be better to use ANN rather than exact attention at inference time. I would imagine this would only pay off with ~100,000 token contexts, and even then it would only work if only a few tokens meaningfully contribute to each attention head.