3 ms·
In this case yeah. If it’s not reading the cache then it has to compute all the context window again and not just the newest tokens.
by vorticalbox 1mo ago
In this case yeah. If it’s not reading the cache then it has to compute all the context window again and not just the newest tokens.