4 ms·
A simple microservice would be https://github.com/huggingface/text-generation-inference https://github.com/huggingface/text-generation-inference . Works flawle
by LoganDark 3y ago
A simple microservice would be https://github.com/huggingface/text-generation-inference https://github.com/huggingface/text-generation-inference .
Works flawlessly in Docker on my Windows machine, which is quite shocking.
Supports Mistral as well as everything else.
Biggest downside is that there's no way to operate the tokenizer through the API. I put in a feature request but they said "you really ought to write your own specialized client-side code for that". Real bummer when the server already supports everything, but oh well.
It has token streaming, automatically halts inference on connection close, and other niceties.
Quite long startup time, but worth it as it doesn't have to be restarted with the client.
- replwoacause 3y agoWhat kind of resources do you need to run this setup, and how well does yours perform? Is it like chatting with any other chat bot (ChatGPT or Claude, for example) or is it significantly slower? Can you train it on your own self-hosted documents, like markdown?
- LoganDark 3y agoThe inference server doesn't do training. The speed is pretty decent for https://huggingface.co/TheBloke/storytime-13B-GPTQ https://huggingface.co/TheBloke/storytime-13B-GPTQ on my 3060, it definitely doesn't feel like you are really waiting for a response. My exact invocation (on Windows) was: $ docker run --gpus all --shm-size 1g -p 8080:80 -v C:\text-generation-webui\models:/data ghcr.io/huggingface/text-generation-inference:1.1.0 --model-id /data/storytime-13b-GPTQ --quantize gptq
- replwoacause 3y agoThanks! I don't have a GPU though, so I'm assuming it isn't going to perform very well. I'll have to see if there are any models that can run on CPU only.