4 ms·
Most likely still 32k tokens under the hood, but with some context slicing/averaging hacks to make inference not error out on infinite input. (That's what I do
by otabdeveloper4 1y ago
Most likely still 32k tokens under the hood, but with some context slicing/averaging hacks to make inference not error out on infinite input.
(That's what I do locally with llama.cpp)