4 ms·
Interesting choice to call llama.cpp directly, instead of relying on a server like Ollama. Nice! I wrote a similar library which calls Ollama (or OpenAI, Verte
by PhilippGille 2y ago
Interesting choice to call llama.cpp directly, instead of relying on a server like Ollama. Nice!
I wrote a similar library which calls Ollama (or OpenAI, Vertex AI, Cohere, ...), with one benefit being zero library dependencies: https://github.com/philippgille/chromem-go https://github.com/philippgille/chromem-go
- milansuk 2y agoNo need to use Ollama. LLama.cpp has its own OpenAI-compatible server[0] and it works great. [0] https://github.com/ggerganov/llama.cpp#web-server https://github.com/ggerganov/llama.cpp#web-server
- PhilippGille 2y agoYes I'm aware. I was contrasting the general use of an inference server vs calling llama.cpp directly (not via HTTP request). And among servers Ollama seems to be more popular, so it's worth mentioning when talking about support for local LLMs.
- citizenpaul 2y agoThanks didn't know that. Do you happen to know the reason to use ollama rather than the built in server? How much work is required to get similar functionality? looks like just downloading the models? I find it odd that ollama took off so quickly if LLamma.cpp had the same built in functionality.