Ollama has built in support [1] for gguf models on huggingface, and exposes a openai compatible http endpoint [2]. You can also just test it out using the cli:

Ollama has built in support [1] for gguf models on huggingface, and exposes a openai compatible http endpoint [2]. You can also just test it out using the cli: ollama run hf.co/unsloth/SmolLM2-1.7B-Instruct-GGUF:F16 1. https://huggingface.co/docs/hub/ollama https://huggingface.co/docs/hub/ollama 2. https://github.com/ollama/ollama?tab=readme-ov-file#start-ollama https://github.com/ollama/ollama?tab=readme-ov-file#start-ol...