3 ms·
Part of the reason for this is likely due to customers preference to have CUDA available which TPUs do not support. TPU is superior for many use cases but custo
by bitexploder 1y ago
Part of the reason for this is likely due to customers preference to have CUDA available which TPUs do not support. TPU is superior for many use cases but customers like the portability of targeting CUDA
- alienthrowaway 1y agoWhat are the pros of using CUDA-enabled devices for inference?
- bitexploder 1y agoMy limited understanding is that CUDA wins on smaller batches and jobs but TPU wins on larger jobs. It is just easier to use and better at typical small workloads. At some point for bigger ML loads and inference TPU starts making sense.
- j5r5myk 1y agoWhich use cases are TPUs superior for?
- riwsky 1y agoRunning Gemini models, for one.