3 ms·
Unfortunately, I think the context rot paper [1] found that the performance degradation when context increased still occurred in models using attention sinks.
by mamp 1y ago
Unfortunately, I think the context rot paper [1] found that the performance degradation when context increased still occurred in models using attention sinks.
1. https://research.trychroma.com/context-rot https://research.trychroma.com/context-rot
- giancarlostoro 1y agoSaw that paper have not had a chance to read it yet, are there other techniques that help then? I assume theres a few different ones used.