Why not? They charge 8x more for a 32k context than for a 8k context (note that the prices are normally presented per token, here I'm talking about absolute cos

Why not? They charge 8x more for a 32k context than for a 8k context (note that the prices are normally presented per token, here I'm talking about absolute cost). Naive scaling on the self-attention component would suggest 16x compute and 4x memory, while the rest of it (forward layers, embeddings, activation functions etc.) all would go up 4x both compute and memory.