3 ms·
But was that with batching? It makes a big difference. You can run many requests in parallel on the same card if you're doing LLM inferencing.
by mike_hearn 1y ago
But was that with batching? It makes a big difference. You can run many requests in parallel on the same card if you're doing LLM inferencing.