LocalAI https://localai.io/ https://localai.io/ and LMStudio https://lmstudio.ai/ https://lmstudio.ai/ both have fairly complete OpenAI compatibility layers. ll

LocalAI https://localai.io/ https://localai.io/ and LMStudio https://lmstudio.ai/ https://lmstudio.ai/ both have fairly complete OpenAI compatibility layers. llama-cpp-python has a FastAPI server as well: https://github.com/abetlen/llama-cpp-python/blob/main/llama_cpp/server/app.py https://github.com/abetlen/llama-cpp-python/blob/main/llama_... (as of this moment it hasn't merged GGUF update yet though)