3 ms·
Does the context size really matter that much if a “dumb zone” starts around 120k-150k anyway?
by fandorin 3mo ago
Does the context size really matter that much if a “dumb zone” starts around 120k-150k anyway?
- conradkay 3mo agoThings change fast! For Fable 5 it definitely feels past at least 272k
- fandorin 3mo agohmm, what about other models?
- wonnage 2mo agoNot sure if this is still up to date (2023), but https://arxiv.org/abs/2307.03172 https://arxiv.org/abs/2307.03172 shows that performance degrades mostly in the middle of the context. Anecdotally I've been stuck in that situation of being at 400-500k tokens and "just one more prompt bro" will get the task done, and I appreciate not having to wait through a compaction. If anything, keeping the bloated context helps with accuracy at the expense of speed in these cases.