I built a simple, asynchronous, serialized API on top of llama.cpp for exactly this reason. https://github.com/edfletcher/llama.http/tree/master/examples/simple

I built a simple, asynchronous, serialized API on top of llama.cpp for exactly this reason. https://github.com/edfletcher/llama.http/tree/master/examples/simple-http https://github.com/edfletcher/llama.http/tree/master/example... It can run on low-resource VPSes even, if you have the patience for CPU inference (which will take awhile)!