3 ms·
Yes, I am so curious how they did it, I know about flash attention but there is no way this gets us all the way there.
by lysecret 4y ago
Yes, I am so curious how they did it, I know about flash attention but there is no way this gets us all the way there.
- sebzim4500 4y agoWhy not? They charge 8x more for a 32k context than for a 8k context (note that the prices are normally presented per token, here I'm talking about absolute cost). Naive scaling on the self-attention component would suggest 16x compute and 4x memory, while the rest of it (forward layers, embeddings, activation functions etc.) all would go up 4x both compute and memory.