4 ms·
What do you use to host these models (like Vicuna, Dolly etc) on your own server and expose them using HTTP REST API? Is there an Heroku-like for LLM models?
by marcopicentini 3y ago
What do you use to host these models (like Vicuna, Dolly etc) on your own server and expose them using HTTP REST API?
Is there an Heroku-like for LLM models?
I am looking for an open source models to do text summarization. Open AI is too expensive for my use case because I need to pass lots of tokens.
- inhumantsar 3y agoWeights and Biases is good for building/training models and Lambda Labs is a cloud provider for AI workloads. Lambda will only get you up to running the model though. You would still need to overlay some job management on top of that. I've heard Run.AI is good on that front but I haven't tried.
- month13 3y agohttps://bellard.org/ts_server/ https://bellard.org/ts_server/ may be what you are after. You can run open-source models, but the software itself is closed-source and free for non-commercial use.
- rain1 3y agoI haven't tried that but https://github.com/abetlen/llama-cpp-python https://github.com/abetlen/llama-cpp-python and https://github.com/r2d4/openlm https://github.com/r2d4/openlm exists
- speedgoose 3y agoThese days I use FastChat: https://github.com/lm-sys/FastChat https://github.com/lm-sys/FastChat It’s not based on llama.cpp but huggingface transformers but can also run on CPU. It works well, can be distributed and very conveniently provide the same REST API than OpenAI GPT.
- itake 3y agoDo you know how well it performs compared to llama.cpp?
- rain1 3y agomy understanding is that the engine used (pytorch transformers library) is still faster than llama.cpp with 100% of layers running on the GPU.
- itake 3y agoI only have an m1
- rain1 3y agoI don't think the integrated GPU on that supports CUDA. So you will need to use CPU mode only.
- itake 3y agoYep, but isn’t there an integrated ML chip that makes it faster than cpu? Or does llama.cpp not use that?
- rain1 3y agounfortunately that chip is proprietary and undocumented, it's very difficult for open source programs to make use of. I think there is some reverse engineering work being done but it's not complete.
- qeternity 3y agoIt's the Huggingface transformers library which is implemented in pytorch. In terms of speed, yes running fp16 will indeed be faster with vanilla gpu setup. However most people are running 4bit quantized versions, and the GPU quantization landscape as been a mess (GPTQ-for-llama project). llama.cpp has taken a totally different approach, and it looks like they are currently able to match native GPU perf via cuBLAS with much less effort and brittleness.