llama.cpp supports multi gpu across local network https://www.reddit.com/r/LocalLLaMA/comments/1cyzi9e/llamacpp_now_supports_distributed_inference/ https://www.

llama.cpp supports multi gpu across local network https://www.reddit.com/r/LocalLLaMA/comments/1cyzi9e/llamacpp_now_supports_distributed_inference/ https://www.reddit.com/r/LocalLLaMA/comments/1cyzi9e/llamacp... and expose an OpenAI compatible server, or you can use their python bindings