You can cobble together an OpenAI-esque server locally with llama.cpp: https://github.com/ggerganov/llama.cpp/issues/2766 https://github.com/ggerganov/llama.cpp

You can cobble together an OpenAI-esque server locally with llama.cpp: https://github.com/ggerganov/llama.cpp/issues/2766 https://github.com/ggerganov/llama.cpp/issues/2766 and the script here: https://www.reddit.com/r/LocalLLaMA/comments/15ak5k4/short_guide_to_hosting_your_own_llamacpp_openai/ https://www.reddit.com/r/LocalLLaMA/comments/15ak5k4/short_g... and use Continue's GGML Model option to query it: https://continue.dev/docs/customization#local-models-with-ggml https://continue.dev/docs/customization#local-models-with-gg... (I've gotten this working, more or less, but don't have the hardware to make it practical so I can't give any feedback about Continue or the model.)