3 ms·
Llama.http: A Dockerized, simple HTTP API for llama.cpp
- edfletcher_t137 3y agoSimple & straightforward, it's a quick spare-time project designed to enable automation and not as a standalone way to publicly host the models (no rate limiting, persistence or auth). The project README is here: https://github.com/edfletcher/llama.http/tree/master/examples/simple-http https://github.com/edfletcher/llama.http/tree/master/example... With the Docker image, it's just a matter of grabbing the model binaries you want, adding a small sidecar JSON [1] describing them and you're up and running. Really just a porting/re-targeting onto the parent llama.cpp project of the HTTP I'd built on the alpaca.cpp fork awhile back [2] in order to take advantage of all the great new GGMLv3-encoded models, mmap'ed loading, etc. All I'd ask if you build anything neat on top of it, please share it so I can use it too! Thanks & enjoy [1] https://github.com/edfletcher/llama.http/tree/master/examples/simple-http#model-sidecar-json https://github.com/edfletcher/llama.http/tree/master/example... [2] https://news.ycombinator.com/item?id=35417730 https://news.ycombinator.com/item?id=35417730