4 ms·
Yes, that is indeed O(N^2). Which, by the way, is not exponential. Also by the way, caching does not make LLM inference linear. It's still quadratic, but the c
by atq2119 6mo ago
Yes, that is indeed O(N^2). Which, by the way, is not exponential.
Also by the way, caching does not make LLM inference linear. It's still quadratic, but the constant in front of the quadratic term becomes a lot smaller.
- computably 6mo ago> Also by the way, caching does not make LLM inference linear. It's still quadratic, but the constant in front of the quadratic term becomes a lot smaller. Touché. Still, to a reasonable approximation, caching makes the dominant term linear, or equiv, linearly scales the expensive bits.