3 ms·
How does it compare to Nvidia A100?
by DeathArrow 3y ago
How does it compare to Nvidia A100?
- ZeroCool2u 3y agoI'm sure they've done benchmarking comparisons with the A100 and H100, but since they're not sharing them it seems unlikely they show the TPU's in a favorable light. Plus, they pick up more spend every time someone runs these benchmarks for themselves.
- deleted 3y ago[deleted]
- RicoElectrico 3y agoThe most important reason all the Big Tech companies spin their own AI chips is that NVIDIA margins are so insane. A purchase from NVIDIA at the hyperscaler volume is on the same order of magnitude as spinning your own chip.
- terafo 3y agov4 chips are basically the same in bf16 performance as A100(slower in int8).
- modeless 3y agoGenerally TPUs don't beat Nvidia per chip. The real reason Google likes them is they are much cheaper in TCO for Google. Of course they may not decide to pass on all the savings to you.
- ithkuil 3y agoI wonder why they don't try to compete as much as they can.
- andromeduck 3y agoThey do - it's just a different design point.
- EvgeniyZh 3y agoIndirectly, using recent 50k tpuv5e run info [1], I'd guess it will give 50-70% of H100 in large-scale LLM jobs: MLCommons show that H100 give a bit more than 500 TFLOPs in FP8, and v5e gives 100 TOPs in INT8. v5p has 2.3 times more theoretical OPs, 3 times more interchip bandwidth and 3.3 times more memory bandwidth, so assuming you can extrapolate and given that bandwidth is usually a bottleneck, ~60% seems plausible. [1] https://cloud.google.com/blog/products/compute/the-worlds-largest-distributed-llm-training-job-on-tpu-v5e https://cloud.google.com/blog/products/compute/the-worlds-la... [2] https://github.com/mlcommons/training_results_v3.1 https://github.com/mlcommons/training_results_v3.1