4 ms·
llama.cpp is optimized to serve one request at a time. vllm is optimized to serve many requests at one time. If you were to fine tune a model and wanted to se
by jasonjmcghee 1y ago
llama.cpp is optimized to serve one request at a time.
vllm is optimized to serve many requests at one time.
If you were to fine tune a model and wanted to serve it to many users, you would use vllm, not llama.cpp
- jasonjmcghee 1y agoHere's a super relevant comment from another post https://news.ycombinator.com/item?id=44366418 https://news.ycombinator.com/item?id=44366418
- barrenko 1y agoAppreciate it!