3 ms·
As the error via linear approximation approaches similar magnitude as numerical error via quadratic computation, don’t the two start becoming comparable in prac
by fheinsen 8mo ago
As the error via linear approximation approaches similar magnitude as numerical error via quadratic computation, don’t the two start becoming comparable in practice?
I ask because in practice, for inference, attention is typically computed with low-precision (4-bit, 8-bit, 16-bit) floats.
Numerical error, in fact, may be a key factor as to why quadratic attention, in practice, exhibits context rot as context gets longer, analogous to an RNN:
https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents https://www.anthropic.com/engineering/effective-context-engi...
- cubefox 8mo agoThat website says nothing about numerical error potentially causing context rot.
- fheinsen 8mo agoAs far as I know, there is no widely accepted explanation for context rot. Numerical error in long sequences of query-key dot-products may be a key factor.
- cubefox 8mo agoThat should be easy to test: test a 16 bit model on various benchmarks, once with fresh context and once with the context filled up with irrelevant tokens. Record the relative performance degradation, and then do the same for a quantized model. Compare whether the quantized model has a significant relatively larger performance drop from context rot. If so, numerical error should be the cause.