4 ms·

This is very neat, thanks for sharing. I was wondering about a related thing — is there a way to query a llama.cpp (or other such local model) via an API from P

This is very neat, thanks for sharing. I was wondering about a related thing — is there a way to query a llama.cpp (or other such local model) via an API from Python? In other words, I see a lot of cool applications being built with langchain + ClosedAPI, so I’m wondering if an API call to a local model could be a drop-in replacement for the ClosedAPI call?

nl 4y ago
There's a llama.cpp fork (I think?) with a build in HTTP server for an API. I'm on mobile and can't find it right now though.
mark_l_watson 4y ago
I have a short example in my recently published book [1] on downloading a HuggingFace model and using it locally with LangChain. [1] https://leanpub.com/langchain https://leanpub.com/langchain EDIT: GitHub repo https://github.com/mark-watson/langchain-book-examples https://github.com/mark-watson/langchain-book-examples
MacsHeadroom 4y ago
Yes, there are python bindings for llama.cpp and the text-generation-webui already uses them for local inference. https://github.com/oobabooga/text-generation-webui/wiki/llama.cpp-models https://github.com/oobabooga/text-generation-webui/wiki/llam... "pip install llamacpp" or https://github.com/thomasantony/llamacpp-python https://github.com/thomasantony/llamacpp-python
boppo1 4y ago
I think langchain lets you do this.
DougBTX 4y ago
In principle you can use subprocess.run with llama.cpp (especially now that the mmap patch has landed, so model load time is negligible on subsequent runs) and then use stdin and stdout to interact with it. Multiple local sessions should work too. I’ve not tried it out yet... though I was looking for an afternoon project to work on!
versteegen 4y ago
Maybe TextSynth server? It has a REST JSON API to pretty much every major LLM running locally with minimal dependencies (no python/pytorch/CUDA). (And I see that since last week it now has an HTML GUI too.) https://bellard.org/ts_server/ https://bellard.org/ts_server/ However, the GPU version is only available commercially. I'd like to see someone compare the speed of the CPU version against PyTorch or llama.cpp. (Edit: llama.cpp's author wrote "I expect LibNC [used by ts_server] will be better in every aspect: performance, accuracy, determinism. But hopefully with time we will close the gap." [1]) EDIT: But if you wish, here's a Python interface to llama.cpp: https://github.com/PotatoSpudowski/fastLLaMa https://github.com/PotatoSpudowski/fastLLaMa [1] https://news.ycombinator.com/context?id=35195270 https://news.ycombinator.com/context?id=35195270
- generalizations 4y ago
  Textsynth is awesome, if you don’t want to use a fine tuned version of the models…far as I can tell, there’s no documentation on how to convert a fine tuned version of a model (like alpaca) to work with textsynth.
- tarruda 4y ago
  I did try llama 30b Q4 on textsynth/CPU. I did not measure, but TextSynth seemed faster than llama.cpp

nl 4y ago

There's a llama.cpp fork (I think?) with a build in HTTP server for an API. I'm on mobile and can't find it right now though.

mark_l_watson 4y ago

I have a short example in my recently published book [1] on downloading a HuggingFace model and using it locally with LangChain. [1] https://leanpub.com/langchain https://leanpub.com/langchain EDIT: GitHub repo https://github.com/mark-watson/langchain-book-examples https://github.com/mark-watson/langchain-book-examples

MacsHeadroom 4y ago

Yes, there are python bindings for llama.cpp and the text-generation-webui already uses them for local inference. https://github.com/oobabooga/text-generation-webui/wiki/llama.cpp-models https://github.com/oobabooga/text-generation-webui/wiki/llam... "pip install llamacpp" or https://github.com/thomasantony/llamacpp-python https://github.com/thomasantony/llamacpp-python

boppo1 4y ago

I think langchain lets you do this.

DougBTX 4y ago

In principle you can use subprocess.run with llama.cpp (especially now that the mmap patch has landed, so model load time is negligible on subsequent runs) and then use stdin and stdout to interact with it. Multiple local sessions should work too. I’ve not tried it out yet... though I was looking for an afternoon project to work on!

versteegen 4y ago

Maybe TextSynth server? It has a REST JSON API to pretty much every major LLM running locally with minimal dependencies (no python/pytorch/CUDA). (And I see that since last week it now has an HTML GUI too.) https://bellard.org/ts_server/ https://bellard.org/ts_server/ However, the GPU version is only available commercially. I'd like to see someone compare the speed of the CPU version against PyTorch or llama.cpp. (Edit: llama.cpp's author wrote "I expect LibNC [used by ts_server] will be better in every aspect: performance, accuracy, determinism. But hopefully with time we will close the gap." [1]) EDIT: But if you wish, here's a Python interface to llama.cpp: https://github.com/PotatoSpudowski/fastLLaMa https://github.com/PotatoSpudowski/fastLLaMa [1] https://news.ycombinator.com/context?id=35195270 https://news.ycombinator.com/context?id=35195270

generalizations 4y ago
Textsynth is awesome, if you don’t want to use a fine tuned version of the models…far as I can tell, there’s no documentation on how to convert a fine tuned version of a model (like alpaca) to work with textsynth.
tarruda 4y ago
I did try llama 30b Q4 on textsynth/CPU. I did not measure, but TextSynth seemed faster than llama.cpp