3 ms·
This has been my experience as well with TPUs and A100s. I haven’t used H100s yet (OOM on 1) but I believe the training throughout benchmarks from Nvidia on tra
by haldujai 3y ago
This has been my experience as well with TPUs and A100s. I haven’t used H100s yet (OOM on 1) but I believe the training throughout benchmarks from Nvidia on transformer workloads is 2.5x from A100s.
The effort to make (PyTorch) code run on TPUs is not worth it and my lab would rather rent (discounted) Nvidia GPUs than use free TRC credits we have at the moment. Additionally, at least in Jan 2023 when I last tried this, PyTorch XLA had a significant reduction in throughput so to really take advantage you would probably need to convert to Jax/TF which are used internally at Google and better supported.