4 ms·
IIUC, the difference is that the TPU allows insane data parallelism - exactly the giant batch sizes that rryan mentions. Google carried out extensive research o
by sdenton4 8y ago
IIUC, the difference is that the TPU allows insane data parallelism - exactly the giant batch sizes that rryan mentions. Google carried out extensive research on batch size, demonstrating that you can greatly increase training efficiency with giant batches. (And with tweaks to optimizers or some regularization, can push the "efficient" batch size domain even further.)
https://ai.googleblog.com/2019/03/measuring-limits-of-data-parallel.html?m=1 https://ai.googleblog.com/2019/03/measuring-limits-of-data-p...
- p1esk 8y agoBatch size is only limited by memory. For comparison Nvidia DGX-2 has 512GB.
- sdenton4 8y agoHere's a writeup on tpu vs got throughput: https://timdettmers.com/2018/10/17/tpus-vs-gpus-for-transformers-bert/ https://timdettmers.com/2018/10/17/tpus-vs-gpus-for-transfor... The underlying principle is that even if the silicon isn't strictly faster, you can shove a dumb amount of data through with the right architecture. And efficient training with data parallelism means that it's a worthwhile strategy.
- p1esk 8y agoI prefer actual benchmark results, which indicate TPU is similar in performance to v100. Unless you have some data that proves otherwise?
- sdenton4 8y agoThis manifests in the training time + cost for imagenet in the stanford DAWN benchmark. The (partial) TPU pods win on training time, with a clear exponential speedup as the pod fraction increases. (How do you use more cores? Increase the batch size.) https://dawn.cs.stanford.edu//2018/04/30/dawnbench-v1-results/ https://dawn.cs.stanford.edu//2018/04/30/dawnbench-v1-result...
- p1esk 8y agoI’m looking at Imagenet training time for Resnet-50 [1] and it appears that 32 V100 chips are faster than half of a TPUv2 pod (64 chips). Are we looking at different benchmarks? [1] https://dawn.cs.stanford.edu//benchmark/ImageNet/train.html https://dawn.cs.stanford.edu//benchmark/ImageNet/train.html
- sdenton4 8y agoNote the submission dates; the faster times have an extra six months.
- p1esk 8y agoHere are more recent results, showing that V100 is competitive with TPUv3: https://www.google.com/amp/s/cloudblog.withgoogle.com/products/ai-machine-learning/mlperf-benchmark-establishes-that-google-cloud-offers-the-most-accessible-scale-for-machine-learning-training https://www.google.com/amp/s/cloudblog.withgoogle.com/produc... I’m having a hard time imagining a task where TPU would be 10x faster than V100.