3 ms·
We’re already using vllm as our inference server for our standard models. We can run whatever inference server for custom deployments.
by FrasiertheLion 1y ago
We’re already using vllm as our inference server for our standard models. We can run whatever inference server for custom deployments.