if you can compile stuff, then looking at llama.cpp (what ollama uses) is also interesting: https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/

if you can compile stuff, then looking at llama.cpp (what ollama uses) is also interesting: https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp the server is here: https://github.com/ggerganov/llama.cpp/tree/master/examples/server https://github.com/ggerganov/llama.cpp/tree/master/examples/... And you can search for any GGUF on huggingface