3 ms·
I thought TPUs are comparable in speed to V100. What makes them 10x faster for your task?
by p1esk 8y ago
I thought TPUs are comparable in speed to V100. What makes them 10x faster for your task?
- sdenton4 8y agoIIUC, the difference is that the TPU allows insane data parallelism - exactly the giant batch sizes that rryan mentions. Google carried out extensive research on batch size, demonstrating that you can greatly increase training efficiency with giant batches. (And with tweaks to optimizers or some regularization, can push the "efficient" batch size domain even further.) https://ai.googleblog.com/2019/03/measuring-limits-of-data-parallel.html?m=1 https://ai.googleblog.com/2019/03/measuring-limits-of-data-p...
- p1esk 8y agoBatch size is only limited by memory. For comparison Nvidia DGX-2 has 512GB.
- sdenton4 8y agoHere's a writeup on tpu vs got throughput: https://timdettmers.com/2018/10/17/tpus-vs-gpus-for-transformers-bert/ https://timdettmers.com/2018/10/17/tpus-vs-gpus-for-transfor... The underlying principle is that even if the silicon isn't strictly faster, you can shove a dumb amount of data through with the right architecture. And efficient training with data parallelism means that it's a worthwhile strategy.
- p1esk 8y agoI prefer actual benchmark results, which indicate TPU is similar in performance to v100. Unless you have some data that proves otherwise?
- sdenton4 8y agoThis manifests in the training time + cost for imagenet in the stanford DAWN benchmark. The (partial) TPU pods win on training time, with a clear exponential speedup as the pod fraction increases. (How do you use more cores? Increase the batch size.) https://dawn.cs.stanford.edu//2018/04/30/dawnbench-v1-results/ https://dawn.cs.stanford.edu//2018/04/30/dawnbench-v1-result...
- p1esk 8y agoI’m looking at Imagenet training time for Resnet-50 [1] and it appears that 32 V100 chips are faster than half of a TPUv2 pod (64 chips). Are we looking at different benchmarks? [1] https://dawn.cs.stanford.edu//benchmark/ImageNet/train.html https://dawn.cs.stanford.edu//benchmark/ImageNet/train.html
- sdenton4 8y agoNote the submission dates; the faster times have an extra six months.
- p1esk 8y agoHere are more recent results, showing that V100 is competitive with TPUv3: https://www.google.com/amp/s/cloudblog.withgoogle.com/products/ai-machine-learning/mlperf-benchmark-establishes-that-google-cloud-offers-the-most-accessible-scale-for-machine-learning-training https://www.google.com/amp/s/cloudblog.withgoogle.com/produc... I’m having a hard time imagining a task where TPU would be 10x faster than V100.