3 ms·
If anyone is interested in avoiding bloat, llama.cpp already includes an OpenAI compatible API: https://github.com/ggerganov/llama.cpp/blob/master/examples/ser
by Art9681 3y ago
If anyone is interested in avoiding bloat, llama.cpp already includes an OpenAI compatible API:
https://github.com/ggerganov/llama.cpp/blob/master/examples/server/api_like_OAI.py https://github.com/ggerganov/llama.cpp/blob/master/examples/...
- brucethemoose2 3y agoThere's also a native C++ implementation now. ...Unfortunately they have issues. The C++ version straight up ignores parameters like temperature, the python implementation does not support batching.