3 ms·
What do you mean? What makes you think that?
by luke-stanley 2y ago
What do you mean? What makes you think that?
- qeternity 2y agoBecause for anything other than CPU inference it is inferior to TensorRT and vLLM.
- luke-stanley 2y agoAre you sure about that? what are the benchmarks it fails on when setup like-for-like with GPU drivers? Even still, it can do constrained grammar and is really easy to setup with wrappers like Ollama and the Python server too. I struggled to find support for that with other inference engines - though things change fast!