3 ms·

1xH100 is faster than 8xA100 for a work-in-progress architecture I'm iterating on. Meanwhile the code doesn't work on TPUs right now because it hangs before ini

1xH100 is faster than 8xA100 for a work-in-progress architecture I'm iterating on. Meanwhile the code doesn't work on TPUs right now because it hangs before initialization. (This is with PyTorch for what it's worth) All that to say Nvidia's hardware work and software evangelization has really paid off-- CUDA just works™ and performance continues to increase. TPUs are good hardware, but TPUs are not available outside of GCP. There's not as much of an incentive for other companies to build software around TPUs like there is with CUDA. The same is likely true of chips like Cerebras' wafer scale accelerators as well. Nvidia's won a stable lead on their competition that's probably not going to disappear for the next 2-5 years and could compound over that time.

nightski 3y ago
It baffles me to this day that Google never made TPUs more widely available. Then again it is Google...
- ls612 3y ago
  They probably saw TPUs as their moat…
haldujai 3y ago
This has been my experience as well with TPUs and A100s. I haven’t used H100s yet (OOM on 1) but I believe the training throughout benchmarks from Nvidia on transformer workloads is 2.5x from A100s. The effort to make (PyTorch) code run on TPUs is not worth it and my lab would rather rent (discounted) Nvidia GPUs than use free TRC credits we have at the moment. Additionally, at least in Jan 2023 when I last tried this, PyTorch XLA had a significant reduction in throughput so to really take advantage you would probably need to convert to Jax/TF which are used internally at Google and better supported.
YetAnotherNick 3y ago
For transformer, v4 chip has 70-100% compute capacity and 40% memory of A100 for pretty much the same price. The only benefit is better networking speed for TPU compared to GPU cluster, allowing very large models to scale better, where for GPU model need to fit in NVlink connected GPUs, which is 320 billion parameters for 8*80 GB A100.
- haldujai 3y ago
  > For transformer, v4 chip has 70-100% compute capacity and 40% memory of A100 for pretty much the same price. Note there are added costs when using V4 nodes such as the VM, storage and logging which can get $$$. > where for GPU model need to fit in NVlink connected GPUs Huh, where is this coming from? You can definitely efficiently scale transformers across multiple servers with parallelism and 1T is entirely feasible if you have the $. Nvidia demonstrated this back in 2021.
  - YetAnotherNick 3y ago
    > Nvidia demonstrated this back in 2021. Because Nvidia created a supercomputer with A100, with lot of focus for networking. Cloud providers don't give that option.
    - haldujai 3y ago
      Azure and AWS have both offered high-bandwidth cluster options that allow scaling beyond a single server for several months now. Pretty sure MosaicML also does this but I haven't used their offering. https://www.amazon.science/blog/scaling-to-trillion-parameter-model-training-on-aws https://www.amazon.science/blog/scaling-to-trillion-paramete...

nightski 3y ago

It baffles me to this day that Google never made TPUs more widely available. Then again it is Google...

ls612 3y ago
They probably saw TPUs as their moat…

haldujai 3y ago

This has been my experience as well with TPUs and A100s. I haven’t used H100s yet (OOM on 1) but I believe the training throughout benchmarks from Nvidia on transformer workloads is 2.5x from A100s. The effort to make (PyTorch) code run on TPUs is not worth it and my lab would rather rent (discounted) Nvidia GPUs than use free TRC credits we have at the moment. Additionally, at least in Jan 2023 when I last tried this, PyTorch XLA had a significant reduction in throughput so to really take advantage you would probably need to convert to Jax/TF which are used internally at Google and better supported.

YetAnotherNick 3y ago

For transformer, v4 chip has 70-100% compute capacity and 40% memory of A100 for pretty much the same price. The only benefit is better networking speed for TPU compared to GPU cluster, allowing very large models to scale better, where for GPU model need to fit in NVlink connected GPUs, which is 320 billion parameters for 8*80 GB A100.

haldujai 3y ago
> For transformer, v4 chip has 70-100% compute capacity and 40% memory of A100 for pretty much the same price. Note there are added costs when using V4 nodes such as the VM, storage and logging which can get $$$. > where for GPU model need to fit in NVlink connected GPUs Huh, where is this coming from? You can definitely efficiently scale transformers across multiple servers with parallelism and 1T is entirely feasible if you have the $. Nvidia demonstrated this back in 2021.
- YetAnotherNick 3y ago
  > Nvidia demonstrated this back in 2021. Because Nvidia created a supercomputer with A100, with lot of focus for networking. Cloud providers don't give that option.
  - haldujai 3y ago
    Azure and AWS have both offered high-bandwidth cluster options that allow scaling beyond a single server for several months now. Pretty sure MosaicML also does this but I haven't used their offering. https://www.amazon.science/blog/scaling-to-trillion-parameter-model-training-on-aws https://www.amazon.science/blog/scaling-to-trillion-paramete...