3 ms·
That website says nothing about numerical error potentially causing context rot.
by cubefox 9mo ago
That website says nothing about numerical error potentially causing context rot.
- fheinsen 9mo agoAs far as I know, there is no widely accepted explanation for context rot. Numerical error in long sequences of query-key dot-products may be a key factor.
- cubefox 9mo agoThat should be easy to test: test a 16 bit model on various benchmarks, once with fresh context and once with the context filled up with irrelevant tokens. Record the relative performance degradation, and then do the same for a quantized model. Compare whether the quantized model has a significant relatively larger performance drop from context rot. If so, numerical error should be the cause.