3 ms·
Here's a writeup on tpu vs got throughput: https://timdettmers.com/2018/10/17/tpus-vs-gpus-for-transformers-bert/ https://timdettmers.com/2018/10/17/tpus-vs-gpu
by sdenton4 8y ago
Here's a writeup on tpu vs got throughput:
https://timdettmers.com/2018/10/17/tpus-vs-gpus-for-transformers-bert/ https://timdettmers.com/2018/10/17/tpus-vs-gpus-for-transfor...
The underlying principle is that even if the silicon isn't strictly faster, you can shove a dumb amount of data through with the right architecture. And efficient training with data parallelism means that it's a worthwhile strategy.
- p1esk 8y agoI prefer actual benchmark results, which indicate TPU is similar in performance to v100. Unless you have some data that proves otherwise?
- sdenton4 8y agoThis manifests in the training time + cost for imagenet in the stanford DAWN benchmark. The (partial) TPU pods win on training time, with a clear exponential speedup as the pod fraction increases. (How do you use more cores? Increase the batch size.) https://dawn.cs.stanford.edu//2018/04/30/dawnbench-v1-results/ https://dawn.cs.stanford.edu//2018/04/30/dawnbench-v1-result...
- p1esk 8y agoI’m looking at Imagenet training time for Resnet-50 [1] and it appears that 32 V100 chips are faster than half of a TPUv2 pod (64 chips). Are we looking at different benchmarks? [1] https://dawn.cs.stanford.edu//benchmark/ImageNet/train.html https://dawn.cs.stanford.edu//benchmark/ImageNet/train.html
- sdenton4 8y agoNote the submission dates; the faster times have an extra six months.
- p1esk 8y agoHere are more recent results, showing that V100 is competitive with TPUv3: https://www.google.com/amp/s/cloudblog.withgoogle.com/products/ai-machine-learning/mlperf-benchmark-establishes-that-google-cloud-offers-the-most-accessible-scale-for-machine-learning-training https://www.google.com/amp/s/cloudblog.withgoogle.com/produc... I’m having a hard time imagining a task where TPU would be 10x faster than V100.