4 ms·
That's interesting. It also looks like koboldcpp works better with long interactions, as it only processes changed tokens. I'm using llama.cpp with text-generat
by rhdunn 3y ago
That's interesting. It also looks like koboldcpp works better with long interactions, as it only processes changed tokens. I'm using llama.cpp with text-generation-webui and its OpenAI compatible API. I'll have to look to see if I can use koboldcpp with it.
- sp332 3y agoLlama.cpp has an interactive mode, but I don't think text-generation-webui uses it. https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md#interaction https://github.com/ggerganov/llama.cpp/blob/master/examples/...
- jodleif 3y agoIndeed. Koboldcpp works fine with other UIs than the bundled one.