3 ms·
Is it more like llama.cpp then? I don't have access to the good hardware.
by barrenko 1y ago
Is it more like llama.cpp then? I don't have access to the good hardware.
- jasonjmcghee 1y agollama.cpp is optimized to serve one request at a time. vllm is optimized to serve many requests at one time. If you were to fine tune a model and wanted to serve it to many users, you would use vllm, not llama.cpp
- jasonjmcghee 1y agoHere's a super relevant comment from another post https://news.ycombinator.com/item?id=44366418 https://news.ycombinator.com/item?id=44366418
- barrenko 1y agoAppreciate it!