Maybe TextSynth server? It has a REST JSON API to pretty much every major LLM running locally with minimal dependencies (no python/pytorch/CUDA). (And I see tha

Maybe TextSynth server? It has a REST JSON API to pretty much every major LLM running locally with minimal dependencies (no python/pytorch/CUDA). (And I see that since last week it now has an HTML GUI too.) https://bellard.org/ts_server/ https://bellard.org/ts_server/ However, the GPU version is only available commercially. I'd like to see someone compare the speed of the CPU version against PyTorch or llama.cpp. (Edit: llama.cpp's author wrote "I expect LibNC [used by ts_server] will be better in every aspect: performance, accuracy, determinism. But hopefully with time we will close the gap." [1]) EDIT: But if you wish, here's a Python interface to llama.cpp: https://github.com/PotatoSpudowski/fastLLaMa https://github.com/PotatoSpudowski/fastLLaMa [1] https://news.ycombinator.com/context?id=35195270 https://news.ycombinator.com/context?id=35195270