3 ms·
So do it locally after predigesting the book, so that you have the entire KV-cache for it. Then load that KV-cache and add your prompt.
by impossiblefork 2y ago
So do it locally after predigesting the book, so that you have the entire KV-cache for it.
Then load that KV-cache and add your prompt.