13 ms·
Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50
- elmarhaussmann 8y agoHi, author here. The motivation for this article came out of the HN discussion on a previous post (https://news.ycombinator.com/item?id=16447096 https://news.ycombinator.com/item?id=16447096). There was a lot of valuable feedback - thanks for that. Happy to answer questions!
- TheLoneAdmin 8y agoAMD - Where does their hardware stand in the race for ML? What changes would AMD need to make to be competitive?
- my123 8y agoTheir hardware is fine. Their software is starting to get good too now. They're finishing MIOpen, a set of CUDA compatible libraries with which you can use Tensorflow (TF uses the builtin CUDA libs too, not only CUDA itself, as does CNTK). ROCm provides a CUDA implementation for AMD systems.
- shaklee3 8y agoTheir hardware doesn't have the equivalent of a tensor core as far as I know, so they would be way behind on these benchmarks.
- MrBuddyCasino 8y agoI found it interesting that they are so close together in performance - I mean what are the odds that they end up within 2% of each other?
- elmarhaussmann 8y agoYeah, pretty big coincidence. However, this may change with the next TensorFlow versions, which supposedly has further speed improvements for the TPUv2. Note also, that the ~2% performance difference is only on one model (ResNet-50) and cannot be generalized to all workloads/all of deep learning (at least not without further proof).
- jacksmith21006 8y agoThe TPUs are doing almost 2x the images for the same cost. That is not all that close is it?
- shaklee3 8y agoNice work. I've only seen anecdotal stories about how TPU is faster, but never something as detailed as this.
- Jabbles 8y agoDo you have more information about this bit? the TPU implementation applies very compute-intensive image pre-processing steps and actually sacrifices raw throughput Thanks
- elmarhaussmann 8y agoIn general, you try to keep the TPU/GPU busy 100%, so enough data needs to be readily accessible at any point in time. In this example, images needs to be read from disk, decoded, transformed (cropped, resized, normalized etc.) before they can be fed to the TPU. The transformations can be computationally intensive so they actually become a bottleneck. In terms of how much compute power the TPU pre-processing needs I only have very rough numbers: I ran the same pre-processing while training ResNet-50 on a node with 4 GPUs and it was consistently utilizing >22 CPU cores (including all of the other CPU-tasks while training).
- puzzle 8y agoDon't TPUs get sustained use discounts? I know they're not preemptible. That would be comparable to AWS reserved instances. EDIT: you don't get sustained use discounts, either, at the moment. You can get either for GCP GPUs, though. Perhaps that will change once TPUs are out of beta?
- sdenton4 8y ago"As shown above, the top-1 accuracy after 90 epochs for the TPU implementation is 0.7% better. This may seem minor, but making improvements at this already very high level is extremely difficult and, depending on the application, such small improvements may make a big difference in the end." Any idea of how much variation in accuracy you get on different training runs of the same model on the same hardware? My understanding is that model quality can and does vary from one run to the next on these kinds of large datasets - from a single observation, it's hard to know if the difference is real or noise.
- antognini 8y agoI've been running a lot of these resnet-50 experiments lately and the run-to-run variation is very small, on the order of 0.1%. It's actually pretty amazing how consistent training is given that the initialization is always different and the data is sampled differently on each run. (As an aside, it took us about three weeks to track down a bug that was causing the model to consistently reach an accuracy 1% lower than it was supposed to.)
- elmarhaussmann 8y agoIndeed, that's also my experience. ImageNet is pretty huge (although 'it's the new MNIST') so that seems to help converging to very similar solutions and accuracies. Tracking down bugs in convergence is really costly in these settings. We had a problem in pre-processing that took us quite a while to figure out...
- pakl 8y agoWhat about your LSTM-based model that didn’t converge in your earlier TPU benchmarks in February?
- kbob 8y agoI am not an ML guy, so I'm asking from a position of ignorance. (-: But what's going on when some of the implementations of a standard algorithm don't converge, and different hardware has different accuracy rates on the same algorithm? Are DNNs really that flaky? And does it really make sense to be doing performance comparisons when the accuracy performance doesn't match? Is the root problem that ResNet-50 works best with a smaller batch size? And how do you do meaningful research into new DNNs if there's always an "Maybe if I ran it again over there I'd get better results" factor? Thank you.
- threeseed 8y agoWas this running the AWS Deep Learning AMI or did you build your own. Because Intel was involved in its development and made a number of tweaks to improve performance. Be curious if it actually was significant or not.
- elmarhaussmann 8y agoOn AWS this was using nvidia-docker with the TensorFlow Docker images. Probably, the AWS AMI Deep Learning gives very similar performance (with same versions of CUDA, TensorFlow etc.). There's only so much you can tweak if the GPU itself is the bottleneck...
- jacksmith21006 8y agoThanks for sharing and very insightful. Guess the TPUs are the real deal. About 1/2 the cost for similar performance. Would assume Google is able to do that because of the less power required. I am actually more curious to get a paper on the new speech NN Google is using. Suppose to be 16k samples a second through a NN is hard to imagine how they did that and was able to roll it out as you would think the cost would be prohibitive. You are ultimately competing with a much less compute heavy solution. https://cloudplatform.googleblog.com/2018/03/introducing-Cloud-Text-to-Speech-powered-by-Deepmind-WaveNet-technology.html https://cloudplatform.googleblog.com/2018/03/introducing-Clo... Suspect this was only possible because of the TPUs. Can't think of anything else where controlling the entire stack including the silicon would be more important than AI applications.
- nojvek 8y agoHalf the cost? Where are you reading that? Yeah on demand rental in AWS is expensive, but both long term and buying V100 yourself is significantly cheaper. Cloud companies have pretty fat margins on on demand rentals. You can’t buy a TPU, it’s a cloud only thing. They also show it’s not a huge difference in both perf and time to converge (albeit only one architecture) I would say kudos to V100 and this benchmark that breaks the TPU hype.
- jacksmith21006 8y agoThe chart has 6.7 per hour for 3186 images Google and 12.2 per hour for 3128 AWS. Or maybe reading it wrong? That is close to half has much to use Google is it not? BTW, The TPUs are also about twice as fast also. Sounds like Google is pretty far ahead of Nvidia. Which really just makes sense as Google does the entire stack and just going to have the data to optimize the silicon. About half the cost is hype? I want in the cloud and not have to deal with updating, etc. Would think most are the same for anything of any scale. Could not imagine any longer building up rigs and dealing with all the issues. Plus much harder to scale.
- nightski 8y ago
- bitL 8y agoExcellent! Thanks for these numbers, I wanted to see exactly this kind of benchmarks! Do you plan to try different benchmarks with the same setup for different problems, like semantic segmentation, DenseNet, LSTM training performance etc. as well?
- elmarhaussmann 8y agoHappy to hear the benchmark is useful to you! We'd love to try different setups and further models/networks. On the other hand, such benchmarks are a LOT of effort (which we underestimated it initially), so we'll have to see.
- wyldfire 8y agoHow much detail do we know about the TPUs' design? Does Google disclose a block-diagram level? ISA details? Do they release a toolchain for low-level programming or only higher-level functions like TensorFlow? EDIT: I found [1] which describes "tensor cores", "vector/matrix units" and HBM interfaces. The design sounds similar in concept to GPUs. Maybe they don't have or need interpolation hw or other GPU features? [1] https://cloud.google.com/tpu/docs/system-architecture https://cloud.google.com/tpu/docs/system-architecture
- elmarhaussmann 8y agoSo far only very few details are disclosed. Here are two presentations: https://supercomputersfordl2017.github.io/Presentations/ImageNetNewMNIST.pdf https://supercomputersfordl2017.github.io/Presentations/Imag... http://learningsys.org/nips17/assets/slides/dean-nips17.pdf http://learningsys.org/nips17/assets/slides/dean-nips17.pdf For the last version of the TPU, Google provided more detail, e.g., in this paper: https://arxiv.org/pdf/1704.04760.pdf https://arxiv.org/pdf/1704.04760.pdf Hopefully, Google will publish something similar for TPUv2, but I have no knowledge whether or when that might happen.
- jacksmith21006 8y agoGreat paper on the Generation 1 TPU. But Google has not shared much details on gen 2 and in some ways kind of hid information. Suspect we will need a gen 3 to get a paper on the gen 2. Here is the gen 1 paper and highly recommend. Pretty interesting using 65536 very simple cores. https://arxiv.org/ftp/arxiv/papers/1704/1704.04760.pdf https://arxiv.org/ftp/arxiv/papers/1704/1704.04760.pdf
- ndesaulniers 8y ago> Maybe they don't have or need interpolation hw or other GPU features? Definitely, no need to do any kind of rasterization here.
- drej 8y agoThanks for this, just a minor thing: You have price per hour and performance per second. Thus that ratio is not performance per image per $, you need to scale that. Also, the metric is not "images per second per $", but just "images per $".
- elmarhaussmann 8y agoThanks for catching this!
- Nokinside 8y agoNvidia is currently in cashing out phase. They have monopoly and money flows in effortlessly. The cost performance ratio reflects this. AMD will enter the game soon once they get their software working, Intel will follow. I suspect that Nvidia will respond with its own specialized machine learning and inference chips to match the cost/performance ratio. As long as Nvidia can maintain high manufacturing volumes and small performance edge, they can still make good profits.
- jacksmith21006 8y ago"The cost performance ratio reflects this." But the TPUs are half the cost per this article? Plus Google does the entire stack and can better optimize the hardware versus Nvidia. So it seem Google can improve faster I would think. If there ever was a huge advantage doing the entire stack it is with neural networks. A perfect example is Google new speech doing 16k samples a second with a NN. https://cloudplatform.googleblog.com/2018/03/introducing-Cloud-Text-to-Speech-powered-by-Deepmind-WaveNet-technology.html https://cloudplatform.googleblog.com/2018/03/introducing-Clo... Do not think Google could offer this service as a competitive cost without the TPUs. This new method is replacing the method that was far less compute intensive so to offer at a competitive price requires lowering compute cost which suspect is only possible with the TPUs.
- shaklee3 8y agoI'm not sure what you mean by google does the entire stack. Nvidia writes all of the major CUDA libraries used behind the scenes in the NN libraries, such as cuDNN, cuBLAS, etc. Nvidia can likely improve their hardware significantly faster/more efficiently than Google can because their entire business depends on it. Google has incentive for improving their TPU for internal use, but they don't make any money by selling TPU time on GCP yet.
- jacksmith21006 8y agoGoogle does the applications at scale and then each layer below and a big one is TF. A great example is the recent release of the new text to speech using NN.
- samfisher83 8y ago>For GPUs, there are further interesting options to consider next to buying. For example, Cirrascale offers monthly rentals of a server with four V100 GPUs for around $7.5k (~$10.3 per hour). However, further benchmarks are required to allow a direct comparison since the hardware differs from that on AWS (type of CPU, memory, NVLink support etc.). Can't you just buy some 1080s for cheaper than this. I understand there is electricity and hosting costs, but cloud computing seems expensive compared to buying equipment.
- elmarhaussmann 8y agoProbably not the best phrasing in the post ("next to buying"). It's only comparing cloud pricing (since the TPUv2 is only available there). If you consider buying hardware the situation is different as you correctly point out.
- dgacmu 8y agoYes, you can. The problem starts when "you" are a large company -- NVidia restricts "datacenter" use of consumer GPUs (see previous HN discussion of that one: https://news.ycombinator.com/item?id=15983587 https://news.ycombinator.com/item?id=15983587 ). A single Titan V is somewhere in the 90% range of a V100 at less than 1/3 the cost, and a 1080ti, if you can find one, likely offers a slightly better price/performance spot. 4-GPU training may suffer due the lack of NVlink, but not enough for it to matter too much. As you scale, though, the lack of NVlink will hurt more. And, of course, all of these things come with a capex vs opex tradeoff, and a sysadmin vs cloud tradeoff, that will appeal differently to different situations.
- twtw 8y agoGreat work, RiseML. This benchmark is sincerely appreciated. I wonder whether NVLink would make any difference for Resnet-50. Does anyone know whether these implementations require any inter-GPU communication?
- elmarhaussmann 8y agoThey don't require it but some of the ResNet-50 implementations can make use of it (e.g., the ones in the Docker containers on the Nvidia GPU Cloud). But even the ones without seem to scale to 4 GPUs pretty well. This may be a different story for 8 GPUs and larger/deeper networks, e.g., ResNet-152.
- Tenoke 8y ago>For the V100 experiments, we used a p3.8xlarge instance (Xeon E5–2686@2.30GHz 16 cores, 244 GB memory, Ubuntu 16.04) on AWS with four V100 GPUs (16 GB of memory each). For the TPU experiments, we used a small n1-standard-4 instance as host (Xeon@2.3GHz two cores, 15 GB memory, Debian 9) for which we provisioned a Cloud TPU (v2–8) consisting of four TPUv2 chips (16 GB of memory each). A bit odd that the TPUs are provisioned on such a weaker machine compared to the V100s, especially when there were comparisons which included augmentation and other processing outside of the TPU.
- elmarhaussmann 8y agoAll of the computation, including pre-processing, is offloaded to the TPU. The weak machine is really just idling. A bigger one will only cost money and have no measurable effect on the performance.
- shaklee3 8y agoThis is why my previous comment mentioned that GCP is a better benchmark for this since you can select the number of CPUs to match with the GPUs to some extent. You can get a rough idea of the savings by looking at their P100 instances.
- KaoruAoiShiho 8y agoWhat is the cost difference between the CPUs on the google cloud vs AWS? How would adjusting for it effect the cost/images ratio?
- deleted 8y ago[deleted]
- puzzle 8y agoThe TPU is not really just the chip. It has an actual machine that is provisioned behind the scenes and accepts RPC calls. Good luck finding out its specs. All you're supposed to care about are the address and port it answers at.
- scottlegrand2 8y agoWhat they're not saying is that one can't use all nvlink bandwidth for gradient reduction on a DGX-1V with only 4 GPUs because nvlink is composed of 2 8-node rings. And given the data parallel nature of this benchmark, I'm very interested in where time was spent on each architecture. That said, they fixed this on NVSwitch so it's just another HW hiccup like int8 was on Pascal.
- elmarhaussmann 8y agoFor this benchmark, NVLink and gradient reduction isn't the bottleneck. The performance scales almost perfectly linearly from one GPU to four.
- zmarty 8y agoSlower alternative: "fastai with @pytorch on @awscloud is currently the fastest to train Imagenet on GPU, fastest on a single machine (faster than Intel-caffe on 64 machines!), and fastest on public infrastructure (faster than @TensorFlow on a TPU!) Big thanks to our students that helped with this." - https://twitter.com/jeremyphoward/status/988852083796291584 https://twitter.com/jeremyphoward/status/988852083796291584
- jorgemf 8y agoOne machine with 8 V100 GPUs. If you consider one TPU pod a single machine the TPU is faster. Those numbers also show that 8 GPUs are slower than 8 TPUs (so same conclusion as the article)
- kyloon 8y agoExcellent work. Do you have plans to open source the scripts/implementation details used to reproduce the results? Would be great if others can also validate and repeat the experiment for future software updates (e.g. TensorFlow 1.8) as I expect there will be some performance gain for both TPU and GPU by CUDA and TensorFlow optimizations. Sidenote: Love the illustrations that accompany most of your blog posts, are they drawn by an in-house artist/designer?
- elmarhaussmann 8y agoHappy you like the post! The implementations we used are open source (we reference the specific revisions), so reproducing results is possible right now. We haven't thought about publishing our small scripts around that (there's not much to it), but it's a good idea. There's also work towards benchmarking suites like DAWNBench (https://dawn.cs.stanford.edu/benchmark/ https://dawn.cs.stanford.edu/benchmark/). The illustrations are from an artist/designer we contract from time to time. I agree, his work is awesome!
- ndesaulniers 8y ago> The illustrations are from an artist/designer we contract from time to time. I agree, his work is awesome! Kudos to them; they are awesome!
- deleted 8y ago[deleted]
- dimitry12 8y agoAn important hidden cost here is coding a model which can take advantage of mixed-precision training. It is not trivial: you have to empirically discover scaling factors for loss functions, at the very least. It's great that there is now wider choice of (pre-trained?) models formulated for mixed-precision training. When I was comparing Titan V (~V100) and 1080ti 5 months ago, I was only able to get 90% increase in forward-pass speed for Titan V (same batch-size), even with mixed-precision. And that was for an attention-heavy model, where I expected Titan V to show its best. Admittedly, I was able to use almost double the batch-size on Titan V, when doing mixed-precision. And Titan V draws half the power of 1080ti too :) At the end my conclusion was: I am not a researcher, I am a practitioner - I want to do transfer learning or just use existing pre-trained models - without tweaking them. For that, tensor cores give no benefit.
- bitL 8y agoHow did you get your hands on Titan V 5 months ago? I still can't find it anywhere in retail in EU...
- dimitry12 8y agoIt was in stock on and off and I was able to order it directly from Nvidia US. After 59 days of playing with it, I sent it back (initiated return on 30th day, after I already figured out it doesn't live up to the hype, then had another 30 days to actually send it back). With $3,000 I can buy 4 1080ti's, while only two are necessary to beat Titan V (in Titan V's best game). I only bought one though. NowInStock.net helped with buying 1080ti directly from Nvidia.
- elmarhaussmann 8y agoAuthor here. Yes, thanks for mentioning that! That's what the article is alluding to at the end. There's also something like a "cost-to-model" and that's influenced by how easy it is to make efficient use of the performance and how much tweaking it needs. It's also influenced by the framework you use... However, that's difficult to compare and almost impossible to measure.
- deleted 8y ago[deleted]