26 ms·
Which GPUs to get for deep learning
- nsriv 6y agoSite down apparently
- rcarmo 6y agoYep. Hugged to death, can’t seem to find a recent mirror/cache either.
- bserge 6y agoFrom just Hacker News traffic? How is that possible, even a (properly configured) Wordpress installation on a Nanode can handle a spike of 10,000+ connections :/
- timdettmers 6y agoIs up again and stable it seems. I shot my cache while trying to "tune" it. Well, I learned my lesson.
- PNWChris 6y agoSame situation for me, luckily it looks like archive.org snagged a backup while it was up! https://web.archive.org/web/20200907164516/https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/ https://web.archive.org/web/20200907164516/https://timdettme... This post looks very thorough, and came just in time for me. I'm looking to snag an upgrade from my GTX 970 for a mix of flight sim 2020 and digging into Fast.ai's course part 2. The 970 has been my big hold-up, right now even simple models take a really long time to work with.
- dgellow 6y agoHave you tried using Google Colab and other online platforms? I started the course a few days ago and so far Colab works well (I don’t really like Jupiter but that’s a detail...). It’s free and you have the choice between a CPU, a GPU, and a TPU.
- PNWChris 6y agoDisclosure (since Colab is a Google product): I work at Google, but everything I say is my personal opinion and experience. I really dig the overall idea of cloud notebooks. Back when I did fast.ai part 1, I used Paperspace Gradient. It was a pretty good experience, but moving files around was a bit of a hassle. For example, getting the images for the Planet Labs exercise took a round trip of downloading from Kaggle to my computer and re-uploading into Jupyter to do analysis. Because of all those moving parts, I decided to give a try running things locally. To my surprise, setup was super easy and I was quickly productive! I really dig how customizable a local Jupyter server is, too. I do use Colab, it's particularly great for collaboration/sharing notebooks, but my past experience has me hooked on the idea of a capable ML machine at home. Plus: I can pitch it to myself and my spouse as an investment in personal development that happens to be able to game :D
- fxtentacle 6y agoGo for it! I have my own Jupyter docker image that I run on a server in the basement. PyCharm can even do code completion for TensorFlow inside a remote docker container. So it's instant, reproducible, and I don't hear a thing in my office :)
- g42gregory 6y agoThank you for posting the archived link. I read it of this archive page, and it's great. This article is so good, that it made to the HN front page without even being available!
- fxtentacle 6y agoSadly, this is mostly purchase advise. In short: You need lots of RAM. And stay away from overclocked (founders edition) and from datacenter models due to heat or price problems.
- timdettmers 6y agoWhat else would you like to see?
- fxtentacle 6y agoThe HN submission was initially called "advice on using GPU for deep learning" so I was hoping for optimization tips, too.
- coredog64 6y agoI’m confused: > Do not buy GTX 16s series cards. These cards do not have tensor cores and, as such, provide relatively poor deep learning performance. I would choose a used RTX 2070 / RTX 2060 / RTX 2060 Super any day over a GTX 16s series card ...a few paragraphs later... > If that is too expensive, a used GTX 980 Ti (6GB $150) or a used GTX 1650 Super ($190).
- KSS42 6y agoHaving a GPU is better than not having one, even if doesn't having tensor cores. If you are on a tight budget, his advice is to pick in this order: > I have little money: Buy used cards. Hierarchy: RTX 2070 ($400), RTX 2060 ($300), GTX 1070 ($220), GTX 1070 Ti ($230), GTX 1650 Super ($190), GTX 980 Ti (6GB $150).
- FridgeSeal 6y agoI guess it should be taken as: * Strongly prefer cards with Tensor cores -> suggested cards * if you need a card because you're currently CPU only, and you're strapped for cash, these are your best bets, but only if you have no other option.
- timdettmers 6y agoThis is good feedback. Will note this down and incorporate it in a small update.
- timdettmers 6y agoI just did a small fix to address this. Thanks again.
- numpad0 6y agoThe intent is probably “as many RTX as new as possible if you’re a full time researcher; otherwise most of recent GTX does fine”, but it’s not explicitly stated in the text.
- aborsy 6y agoThanks for the post, it was very good! It would be great if you could add something about the hardware requirements of the reinforcement learning and video prediction.
- tbenst 6y agoI appreciate the time and care that went into this post, and there’s a nice discussion of various features. Unfortunately the performance charts are completely devoid from reality, and in particular the discussion on tensorcores may be true from an instruction count perspective but does not reflect any third-party benchmark I’ve seen. For example: https://lambdalabs.com/blog/2080-ti-deep-learning-benchmarks/ https://lambdalabs.com/blog/2080-ti-deep-learning-benchmarks.... Nvidia has a history of straight-up-lying about tensorcore and other benchmarks (for example, see this thread from right after Nvidia announced an 8x improvement in speed on imagenet in tensorflow for V100: https://github.com/tensorflow/benchmarks/issues/77 https://github.com/tensorflow/benchmarks/issues/77) In general, fp16 is only 30-40% faster than fp32, and occasionally 2x in really optimal conditions.
- llukas 6y agoCannot really lie here: https://mlperf.org/ https://mlperf.org/
- YetAnotherNick 6y agoI don't see the timing comparison between FP16 and FP32. Can you point me at it if it is there.
- nl 6y ago> Unfortunately the performance charts are completely devoid from reality, and in particular the discussion on tensorcores may be true from an instruction count perspective but does not reflect any third-party benchmark I’ve seen. For example: https://lambdalabs.com/blog/2080-ti-deep-learning-benchmarks/ https://lambdalabs.com/blog/2080-ti-deep-learning-benchmarks... The performance numbers posted here appear to almost exactly reflect the LambdaLabs numbers. Lambda Labs: the RTX 2080 Ti is 96% as fast as Titan V, 73% as fast as Tesla V100 (32 GB) timdettmers: RTX 2080 Ti normalized to 1, Titan V looks about 1.1 to 1.2, V100 is just below 1.5 > for example, see this thread from right after Nvidia announced an 8x improvement in speed on imagenet in tensorflow for V100 Well a non-NVidia person managed to get it up to just above 4x improvement without "using unreleased libraries from NVIDIA". From the same thread: https://github.com/tensorflow/benchmarks/issues/77#issuecomment-394541623 https://github.com/tensorflow/benchmarks/issues/77#issuecomm... In my experience NVidia benchmark numbers in deep learning are rarely lies - they are highly optimised, in optimal conditions and rarely achievable in the real world. About what you'd expect from a vendor benchmark.
- lostdog 6y agoReally great to publish these builds and GPU suggestions. Putting together a system that works can be really frustrating, and knowing where to start is really helpful. I've gone with GPU spot instances for my personal experiments. The key is to be able to bring up a machine in a couple minutes so you're ok with always tearing one down. A combination of ansible and some scripts that push code around helped a lot to create a useful environment for experimenting.
- nl 6y agoJust noting that you can (still) do very well on non top-of-the-line cards. I've won multiple silver Kaggle medals on a 1070. It's true that more power would be helpful, but I feel it's lack of technique (and time!) rather than compute that has held me back from gold medals.
- edude03 6y agoNot disagreeing with you, but from what I've heard, the people who win kaggles are people who can try more things, having a faster card presumably would allow you to try more things because each attempt would take less time.
- nl 6y agoThere is some truth in this, but it's not the entire picture. I've thought about it a lot, and talked to lots of really good Kagglers about it. Most of them use multiple machines, rather than having maximum performance in a single machine. This lets them run multiple completely different experiements at once, and then put extra compute onto the ones that seem good. That is a big difference to me, when I have to experiment sequentially. The parallelism is more important than absolute speed a lot of the time.
- ablekh 6y agoKudos to the author on producing an excellent, comprehensive yet readable, post (based on my initial very brief review)! Much appreciated. One thing that jumped at me, though, is the recommendation for the "I want to try deep learning, but I am not serious about it" scenario. Advising to use a physical GPU (unless it's already a part of the system at hand), especially RTX 2060 Super, IMO does not make any sense in this case. Using cheap cloud GPU instances is the optimal way to try deep learning.
- timdettmers 6y agoThank you! You have a good point. I think I would agree with you if somebody already has cloud computing skills, then the cloud is much more powerful to learn deep learning than your own GPU. I figured that most people that start with deep learning might also lack cloud computing skills. Learning one thing at a time is easier and as such, just sticking a GPU into your desktop and focus on deep learning software / programing might yield a better experience. I might update my blog post in the future with this detail.
- ablekh 6y agoYou're welcome! While I understand your rationale now, I'm afraid I still disagree with it. :-) Simply because I find it very unlikely that people interested in and having enough skills to embark on any reasonable deep learning journey (even if just to try) would lack enough cloud computing skills to use cloud GPU instances. After all, using GPUs in the cloud is not much different from (and, thus, not more complex than) using physical GPUs in your local machine. Fun fact: I've saved your post as PDF for offline reading and it clocked at 649(!) pages at the time of saving (32 pages for the post per se and the rest for the blog post comments). Combining that with feedback here at HN, it is clear that there is quite a lot of interest in the topic ...
- timdettmers 6y agoHaha, 649 pages! Thanks for the discussion. I can understand your perspective. Maybe it would be best to add something to the blog post that discusses it from both perspectives and readers can then choose which perspective suits them better. I should also a bit more in general about cloud computing, it seems some people agree that the post ran a bit short on that. At some point I just wanted to be done with it though — editing 10k word blog posts is not so much fun anymore!
- RedComet 6y agoAre any of the bigger frameworks supporting AMD yet?
- dade_ 6y agoI want an external 3090 chassis on Thunderbolt, similar to this box. No muss, no fuss. AC, liquid cooling, all in an engineered box. Gaming or ML with my laptop or tablet pc. https://www.gigabyte.com/Graphics-Card/GV-N208TIXEB-11GC#kf https://www.gigabyte.com/Graphics-Card/GV-N208TIXEB-11GC#kf
- ablekh 6y agoI'm sure that eGPU enclosure manufacturers will eventually produce models fully compatible with GeForce RTX 30 series. In the meantime, FYI there are some relevant discussions regarding potential compatibility (with pin adapters) of some eGPU enclosures currently available on the market, e.g.: https://egpu.io/forums/thunderbolt-enclosures/razer-core-x-chroma-rtx-3080-12-pin-compatability https://egpu.io/forums/thunderbolt-enclosures/razer-core-x-c....
- numpad0 6y agoJust buy a good ITX PC. eGPU boxes often costs more than a PC, larger than one as well, while performing less than one due to TB3 link and overall clunkiness.
- ChuckNorris89 6y agoThunderbolt 4 is on it's way with Intel Tigerlake which doubles the bandwidth but that still leaves limitations such as being limited to x8 PCI lanes which means you're leaving performance on the table if you use high eng GPUs and being locked in to Intel ecosystem so no Ryzen CPUs for you :( And also cost $$$ which makes it a luxury solution only suitable for the users who absolutely want a thin portable notebook for on the go and a powerful GPU at home for AI/Gaming.
- spicyramen 6y agoMmm I don't see any mention of Cloud offerings or T4 model, specifically Google Cloud team revealed a more detailed and comprehensive blog post which include more important aspects: inference, training, cost and time. https://cloud.google.com/blog/products/ai-machine-learning/your-ml-workloads-cheaper-and-faster-with-the-latest-gpus/ https://cloud.google.com/blog/products/ai-machine-learning/y...
- shmerl 6y agoLooks very Nvidia centric. What about using other GPUs for deep learning?
- nanagojo 6y agoCUDA owns the industry. AMD has to step up with a good alternative to disrupt it. Any serious work is being done in CUDA and there are lots of resources for CUDA.
- sdenton4 6y ago(well, there is also 'serious' work being done with TPUs...)
- shmerl 6y agoYou don't need to use CUDA for GPU programming, no? It's simply a lock-in. But I'm sure Nvidia was pushing it quite lot to make sure it's hard not to use it. But it clearly has to go. It's not healthy for the industry.
- atomlib 6y agoAre you trying to say Nvidia created a great framework for GPU compute so everyone is using it?
- shmerl 6y agoNvidia probably sponsored a lot of high level frameworks that only support CUDA, essentially poisoning them with lock-in.
- atomlib 6y agoNo, it did not. There are plenty of entirely independent benchmarks showcasing Nvidia's advantage. As far as I'm concerned Nvidia has honest and deserved reason to say they currently produce the best GPUs both for both gaming and compute workloads. Are you an unpaid AMD advocate or something? There's nothing inherently “clean” or morally right about buying AMD products.
- tanilama 6y agoFor NLP applications, the deciding factor is actually GPU memory, so the choice is limited (V100 32GB the best, and nothing below 16GB is worth considering)
- p1esk 6y agoQuadro 8000 has 48GB and costs ~$5,300, so it's a much better value than V100.
- liuliu 6y agoGood article! But the comment on NVLink / PCIe4 (and PCIe3 x4 is good enough) doesn’t fit my experience. PCIe3 x4 can really impact your all-reduce performance for models such as Transformers. For ResNet, it can impact a bit too, but to the 5% to 10% range like the author mentioned. I am also interested in whether PCIe4 can help unified memory for larger models. Guess have to wait for RTX 3090 actual release.
- p1esk 6y agounified memory for larger models What are you talking about? Are you writing custom CUDA code to run those large models? Because there's zero support for unified memory in any of the existing DL frameworks.
- liuliu 6y agoYes. I have an alternative DL framework to play whatever I want with :)
- p1esk 6y agoLink?
- lunixbochs 6y agoProbably this: https://libnnc.org https://libnnc.org https://github.com/liuliu/ccv/tree/unstable/lib/nnc https://github.com/liuliu/ccv/tree/unstable/lib/nnc
- lunixbochs 6y agoI agree. When training production speech recognition models (wav2letter) on huge datasets, I hit PCIe, NVLink, network, and probably storage bottlenecks when scaling. (This isn’t comparable to training on 40GB of OpenWebText, it’s terabytes of compressed audio) When I tested the DGX2, 16-card bandwidth was finally not an issue (I was extremely happy with the NVSwitch)... but I hit a CPU bottleneck going from [64vcpu + 8 V100] to [96vcpu + 16 V100] (even with tricks to reduce CUDA CPU load). I’m excited to sometime try my workload on 2x NVLinked 3090s with a 64-core AMD CPU, optimized PCIe v4 NVMe data path, additional caching, and some hand tuning of the training code. It’s possible this will be competitive with an 8x V100 google cloud instance based on some of my scaling pain graphs. I’d consider 4x 3090s- but between only having a single NVLink port, and having to figure out how to even fit four 3x cards in a case without water cooling, it seems prudent to start with two.
- leoh 6y agoWhy not just use a cloud offering? The AWS analysis is interesting, but there are other great offerings like Google's colab, which offers a free GPU https://colab.research.google.com/notebooks/intro.ipynb https://colab.research.google.com/notebooks/intro.ipynb.
- esquire_900 6y agoBought a second hand GTX 1060 a while ago to play around a bit more serious with neural networks. It's a good balance between being cheap, 6GB of memory but still being serious enough to get some work done. If you are a professional researcher or do multiple kaggle's per month, then yes, get the best card. But I suspect a lot of people are one category "below", where a 1060 is sufficient most of the time, and you go to cloud for the actual big workloads. This strategy can keep you going for a couple of years. With models becoming as big as they are, I doubt how much SOTA an RTX 3070 is going to do in 2-3 years. (actually none of these cards come close to GPT-3). By that time you can pick up a second hand RTX 30x and still get the latest offerings in the cloud. Buying second hand GPU comes with a bit of a risk by the way, someone suggested only buying if the price is less than half of the original price (can't remember the link).
- mastazi 6y agoInterestingly, the 1060 was mentioned among the suggested cards in one of the old versions of this same article (it gets updated every time Nvidia releases a new series).
- djaque 6y agoIs anyone else trying to decide if you should upgrade from 1080 ti's? I have a box with four of them which have served me well for a while now, and based on just raw performance it looks like upgrading to two RTX 3080s would exceed the performance of my current system. I'm wondering if I should rush to sell off the cards on the used market before the prices crash and then use that money to swap over to Ampere. Then, there's also the question of if there will be an RTX 3080 TI which will blow away the RTX 3080 and be a viable card for the next five years like the 1080 TI. I'm really uncertain about what to do and wonder the calculus other people have done on this decision.
- SloopJon 6y agoOne thing I did after the RTX 30-series announcement was a back-of-the-envelope comparison of performance per dollar and performance per watt, taking NVIDIA's numbers at face value. The 3070 and 3080 are surprisingly close on both metrics. You pay a substantial premium for the 3090, but it does have the best performance per watt.
- oxygenz 6y agoIt looks like most are thinking 3080 might be the sweetspot for value?
- acidbaseextract 6y agoDefinitely, especially from a memory bandwidth perspective: https://youtu.be/KpnIx1kLq9w https://youtu.be/KpnIx1kLq9w
- zmmmmm 6y agoWould be interested to know how the Tesla T4's fit in, if at all. They seem to be by far the most "affordable" if your goal is to get in at the low end in the data center space. But not at all sure if they represent value in terms of $/compute?
- ptheywood 6y agoT4s are essentially a 2070 super with twice the device memory (plus minor changes to clock speeds to account for power/cooling). ~ 5x the price, but suitable for the data centre and larger networks.
- aqohn123 6y agoAnyone here knows an article comparing GPU architectures for non deep learning that has a similar depth as this article addressing stuff like memory latencies, cycles, caches etc. too? Kudos to the author of above article, I really enjoyed reading it!
- ngcc_hk 6y agoBefore I push that button have to say that I bought two 1080ti based on the site 3 years ago. Still remember the concern on memory. Hope it will not ... 3090 got 24 and price is not as unreachable as Titan then.
- oxygenz 6y agoagreed!
- ngcc_hk 6y agoNo comment on the direct transfer from Ssd to Gpu bypassing cpu ... Amy relevance.
- jtflynnz 6y agoOut of curiosity, there are now enterprise sellers offloading old Tesla cards for relatively cheap (ie K40 ~$100 - $150, k80/M40 ~$150 - $200); are these worth looking at on a budget versus the 900 or 1600 series cards? Especially given memory options.
- Apofis 6y agoAnswer: Whatever your current generation mid-level Nvidia Geforce is. It's been this way for a while. Though, you should probably use AWS ML Compute, since they even have Nvidia Ampere 100's, which cost $10,000 each, and it'll probably be more cost effective for heavier workloads.
- k12sosse 6y agoWish they would make dedicate ML cards so the gamers don't have to fight with the deepfakers and miners