3 ms·
> Also I'm not sure if ollama supports a kv-cache between invocations of /v1/completions, which could help) Not sure about ollama, but llama-server does have a
by tarruda 1y ago
> Also I'm not sure if ollama supports a kv-cache between invocations of /v1/completions, which could help)
Not sure about ollama, but llama-server does have a transparent kv cache.
You can run it with
llama-server -hf ggml-org/gpt-oss-20b-GGUF -c 0 -fa --jinja --reasoning-format none
Web UI at http://localhost:8080 http://localhost:8080 (also OpenAI compatible API)