In principle you can use subprocess.run with llama.cpp (especially now that the mmap patch has landed, so model load time is negligible on subsequent runs) and

In principle you can use subprocess.run with llama.cpp (especially now that the mmap patch has landed, so model load time is negligible on subsequent runs) and then use stdin and stdout to interact with it. Multiple local sessions should work too. I’ve not tried it out yet... though I was looking for an afternoon project to work on!