4 ms·
Nobody is running llama.cpp in production…
by qeternity 2y ago
Nobody is running llama.cpp in production…
- luke-stanley 2y agoWhat do you mean? What makes you think that?
- qeternity 2y agoBecause for anything other than CPU inference it is inferior to TensorRT and vLLM.
- luke-stanley 2y agoAre you sure about that? what are the benchmarks it fails on when setup like-for-like with GPU drivers? Even still, it can do constrained grammar and is really easy to setup with wrappers like Ollama and the Python server too. I struggled to find support for that with other inference engines - though things change fast!
- szundi 2y agoProbably more often than not
- qeternity 2y agoHighly doubtful. If anyone is running it here in a production setting, please post and prove me wrong.