elmarhaussmann

6 ms·

1.
by elmarhaussmann 8y ago
Author here. Yes, thanks for mentioning that! That's what the article is alluding to at the end. There's also something like a "cost-to-model" and that's influenced by how easy it is to make efficient use of the per
2.
by elmarhaussmann 8y ago
For this benchmark, NVLink and gradient reduction isn't the bottleneck. The performance scales almost perfectly linearly from one GPU to four.
3.
by elmarhaussmann 8y ago
Happy you like the post! The implementations we used are open source (we reference the specific revisions), so reproducing results is possible right now. We haven't thought about publishing our small scripts around that (there's n
4.
by elmarhaussmann 8y ago
Indeed, that's also my experience. ImageNet is pretty huge (although 'it's the new MNIST') so that seems to help converging to very similar solutions and accuracies. Tracking down bugs in convergence is really costly in
5.
by elmarhaussmann 8y ago
In general, you try to keep the TPU/GPU busy 100%, so enough data needs to be readily accessible at any point in time. In this example, images needs to be read from disk, decoded, transformed (cropped, resized, normalized etc.) before
6.
by elmarhaussmann 8y ago
Thanks for catching this!
7.
by elmarhaussmann 8y ago
All of the computation, including pre-processing, is offloaded to the TPU. The weak machine is really just idling. A bigger one will only cost money and have no measurable effect on the performance.
8.
by elmarhaussmann 8y ago
They don't require it but some of the ResNet-50 implementations can make use of it (e.g., the ones in the Docker containers on the Nvidia GPU Cloud). But even the ones without seem to scale to 4 GPUs pretty well. This may be a differen
9.
by elmarhaussmann 8y ago
Probably not the best phrasing in the post ("next to buying"). It's only comparing cloud pricing (since the TPUv2 is only available there). If you consider buying hardware the situation is different as you correctly point out
10.
by elmarhaussmann 8y ago
Yeah, pretty big coincidence. However, this may change with the next TensorFlow versions, which supposedly has further speed improvements for the TPUv2. Note also, that the ~2% performance difference is only on one model (ResNet-50) and can
11.
by elmarhaussmann 8y ago
Happy to hear the benchmark is useful to you! We'd love to try different setups and further models/networks. On the other hand, such benchmarks are a LOT of effort (which we underestimated it initially), so we'll have to see.
12.
by elmarhaussmann 8y ago
So far only very few details are disclosed. Here are two presentations: https://supercomputersfordl2017.github.io/Presentations/Imag... http://learningsys.org/nips17/assets/slides/dean-ni
13.
by elmarhaussmann 8y ago
On AWS this was using nvidia-docker with the TensorFlow Docker images. Probably, the AWS AMI Deep Learning gives very similar performance (with same versions of CUDA, TensorFlow etc.). There's only so much you can tweak if the GPU itse
14.
by elmarhaussmann 8y ago
Hi, author here. The motivation for this article came out of the HN discussion on a previous post ( https://news.ycombinator.com/item?id=16447096 ). There was a lot of valuable feedback - thanks for that. Happy to answer ques
15.
by elmarhaussmann 9y ago
Thank you for your feedback! (author here) Our intention is really to provide a sound comparison. I think we agree that these kinds of comparisons can be hard given the constraints (e.g., lack of available technical information on TPUv2 or
16.
by elmarhaussmann 9y ago
It was my understanding that the TensorFlow benchmarks do make use of TensorCores on the V100. We'll verify and update accordingly.
17.
by elmarhaussmann 9y ago
I tested many different batch sizes for the LSTM, so I am pretty confident it's not the reason.
18.
by elmarhaussmann 9y ago
Author here. Thanks for your feedback. As I noted above, we will report further results with larger batch sizes (and smaller ones for the TPU). The LSTM not converging is one of the experiences we wanted to share. We are working on solving
19.
by elmarhaussmann 9y ago
Author here. These are only available on the Google Cloud right now. I don't think there are plans to sell them anytime soon.
20.
by elmarhaussmann 9y ago
Author here. No, this is really only comparing the throughput on the devices. A thorough comparison should really focus on time to reach a certain quality - including all of the tricks available for a certain architecture.
21.
by elmarhaussmann 9y ago
Author here. Note that the TPU supports larger batch sizes because it has more RAM. We tested multiple batch sizes for GPUs and reported the fastest one. We'll try increasing the batch sizes as far as possible and report. The overall c
22.
by elmarhaussmann 9y ago
Author here. Point well taken, we'll make sure to add a comparison to 4 and 8 GPUs. For now, a "Cloud TPU" (containing 8 cores) seems to be the smallest unit to allocate. The question of what exactly makes up a single device
23.
by elmarhaussmann 9y ago
Would you expect a big performance difference from using CUDA 9.1?
24.
by elmarhaussmann 9y ago
Author here. Good point, I agree that the FP16 GPU results should be closer or grouped with the TPU results. We'll try to update accordingly.
25.
by elmarhaussmann 9y ago
Author here. Thanks for your feedback and your suggestions (and from everybody else)! We'll make sure to gather all of the valuable feedback and run additional experiments. Different batch sizes and a comparison against >1 GPUs is a
26.
by elmarhaussmann 9y ago
Where specified "fp16", the V100 benchmarks use the code from https://github.com/tensorflow/benchmarks/tree/master/scripts... with the flag --use_fp16=true which enables fp16 for some but not a