3 ms·
Can you share a source or docs describing the “running binary context”? I am using this stack (as a dev, not an ml engineer) and appreciate the tip but would li
by threecheese 3y ago
Can you share a source or docs describing the “running binary context”? I am using this stack (as a dev, not an ml engineer) and appreciate the tip but would like to understand the delta better. Just spent a few minutes searching through code but don’t see anything obvious. Thanks!
- lolinder 3y agoMy second link documents the `context` parameter for Ollama. It's unfortunately pretty hard to search for because the term is overloaded. `context` is a number array that gets returned with each /generate request and can be returned to the server with the next request. There's not much more documentation than what's there in that link, but as I understand it the array is a JSON representation of the current state of inference at the time that Ollama returned. So when you send the context back up with a second request, Ollama can just load that array into memory and then start inference with the new prompt, rather than having to re-encode the whole conversation up to that point.
- CuriouslyC 3y agoI'm pretty sure it's just a cache of the context embedding.