3 ms·
To run inference on GPUs, people are typically using TensorRT or a similarly-optimized engine. That can make a big difference in cost tradeoffs vs. CPU. Ultimat
by zetazzed 6y ago
To run inference on GPUs, people are typically using TensorRT or a similarly-optimized engine. That can make a big difference in cost tradeoffs vs. CPU. Ultimately, if you can keep a GPU reasonably well-fed, the GPU can come out much cheaper and lower latency. If your workload is very sporadic and infrequent, YMMV.
- pbalau 6y ago> if you can keep a GPU reasonably well-fed That's the big if that most people seem to miss. And I even had people complaining their training was slow on a GPU...