4 ms·
Can it easily run as a server process in the background? To me, not having to load the LLM into memory for every single interaction is a big win of Ollama.
by lxgr 1y ago
Can it easily run as a server process in the background? To me, not having to load the LLM into memory for every single interaction is a big win of Ollama.
- otabdeveloper4 1y agoYes, of course it can.
- lxgr 1y agoI wouldn't consider that a given at all, but apparently there's indeed `llama-server` which looks promising! Then the only thing that's missing seems to be a canonical way for clients to instantiate that, ideally in some OS-native way (systemd, launchcd etc.), and a canonical port that they can connect to.