12 ms·

Which GPU(s) to Get for Deep Learning

166 points by laktak 9y ago

atarian 9y ago
Google cached version: http://webcache.googleusercontent.com/search?q=cache:04az_uBF27EJ:timdettmers.com/2017/04/09/which-gpu-for-deep-learning/ http://webcache.googleusercontent.com/search?q=cache:04az_uB...
- godelski 9y ago
  Thank you, I was getting Error 500 - Hug Of Death
ScottBurson 9y ago
Site is slammed, but I think this is from a couple years ago. -- Ah, it's been updated. Is there anything better now than the GTX 1080 Ti?
- Analemma_ 9y ago
  The Titan Xp is better, but it's not so much better that it's worth almost twice the price.
  - paulsutter 9y ago
    Two 1080TIs are way better than one Titan Xp and cost about the same (2x$700 vs $1200). Each TI has 11GB of RAM vs the TitanX's 12GB, and each TI is nearly as fast. At the NVIDIA conference all the second tier hosting companies were promoting the P100 (at NVIDIAs insistence) but when pressured admitted that their big customers now deploy 1080TIs. Paying for P100s is sort of a clown move even if you're spending someone else's money. The P100 starts at like 10x the price of the 1080TI and isn't much faster and again has just 12GB of RAM (16GB if you pay $4000 more, ludicrous)
    - mattnewton 9y ago
      Usually, but the caveat is they are only better for workloads that are easily paralizable (like hyperparameter search). Multi-gpu models are still very complicated in my experience. And the Xp is a bit faster and has 1gb more memory, which is useful in some edge cases (like large or 3D covnets) Also if there do build with multiple gpus, watch out that you can give both cards a full 16 pci lanes, not a given in a lot of motherboards.
  - sp332 9y ago
    Details https://arstechnica.com/gadgets/2017/04/nvidia-titan-xp-gp-102-details-price/ https://arstechnica.com/gadgets/2017/04/nvidia-titan-xp-gp-1... If your machine learning problem is sized to fit 12GB better than 11GB, it could save a lot of time. But unless you're really in a hurry, it's still probably not worth the huge price bump.
sgt101 9y ago
No mention of k80's k40's...
- chronic940 9y ago
  K80s and K40s have a K. That's two generations old. We are currently on P and about to be V. If you think K80s are good, you are far far behind the times in machine learning. A single 1080Ti even in a 4U server outperforms 2 K80s.
  - sgt101 9y ago
    Ok - p100's? The thing is that a K80 costs.... £4k what is it that people are paying for?
    - dragandj 9y ago
      ECC memory (not that everyone really need that)...
      - sgt101 9y ago
        In a tensorflow job I find it hard to imagine that a memory error would make a jot of difference... wouldn't the error simply be removed by the network training? If the error was in the execution then I imagine that the epoch would fail and it would then just be a matter of restarting.
  - arnon 9y ago
    If you're running a rackmount server, you need the Tesla series. We found that the GeForces tend to burn out when under heavy load, whereas we've not had a single Tesla series ever burn out.
    - maksimum 9y ago
      Ooh ooh tell us more. Which GeForces and which server chassis? Adequate power supply?
ericfrederich 9y ago
I'm guessing you don't... you use the cloud, right?
- exelius 9y ago
  If you're doing a lot of experimentation, it can make more sense to train your models locally (so you're not paying server / data transfer costs every time you test).
- antognini 9y ago
  For a hobbyist, it probably makes sense to just use some GPU instances on AWS. But it doesn't really take that long for it to become cost-effective to buy a GPU. It only takes a couple of months of full-time use before it becomes cost-effective to purchase the GPU. Even if you're building the entire machine from scratch it takes under a year. (At least, that was my estimate when I was building a rig earlier this year.) If you're doing any serious deep learning projects it's pretty easy to have a model training nearly 100% of the time.
  - nightski 9y ago
    Not to mention, I can imagine it's frustrating uploading many gigs of data to amazon.
    - treehau5 9y ago
      extremely frustrating. Internet service is getting worse this in country, not better, sadly. The only lucky ones are those who live in competitive markets (lol) and those who had google fiber break some ground. The anti-competitiveness is getting worse too. Charter just took over TWC in my area.
    - zeptomu 9y ago
      It is not that bad. Either you've your training data local (unlikely) or it is already available in the "cloud" (i.e. a public-facing service). Let's say a typical training set (raw data) is 100G. Assuming 2.5MB/s Upload capacity with local data, you've uploaded it to your deep learning machine in half a day (100000 / 2.5 / 3600 ~ 11 hours) - which is not that much, as most of your time will be used for development and fine-tuning of your deep learning tool chain anyway. In most cases the data is accessible via a public-facing service, and assuming 1GBit/s bandwidth you've downloaded 100G in 13 minutes (100000 / 125 / 60 ~ 13 minutes).
jacquesm 9y ago
tldr: GTX1080ti They're fairly easy to get now too, in the beginning it was rather hard to get them.
- astrodust 9y ago
  You can get an 8GB GTX 1080 for 2/3rds the price and it offers 80% of the performance. If you don't need the 11GB of memory, it's a steal. If you're building a rig with multiple cards, 1080 might be an economical way to get great performance. If you're looking for maximal compute density, 1080 Ti is the way to go.
  - ex3ndr 9y ago
    It will also require much more power and cooling this system will rise a cost and reduce stability of the system.
    - astrodust 9y ago
      The power consumption of a 1080 is actually lower than a 780 or 980, plus a 1080 Ti uses about 50W more power when pinned, so I'm not sure where you're coming from here.
AlphaSite 9y ago
Interestingly, AMD is claiming that their upcoming Vega Frontier Editions chips outperform Nvidia's current P100 Deep Learning chip. http://hexus.net/media/uploaded/2017/5/30f5633b-1bbf-49b7-9f31-7890e20db28f.jpg http://hexus.net/media/uploaded/2017/5/30f5633b-1bbf-49b7-9f...
- sp332 9y ago
  I think signs are pointing to Vega being a HPC beast. But what they really need is the software ecosystem and support, and so far that hasn't been there. So while the new hardware looks cool, I'm really waiting for an announcement that OpenCL tooling got a lot better, or that CUDA is getting first-party support from AMD, anything to tempt those customers away from nvidia.
  - dragandj 9y ago
    CUDA is getting support from AMD in the form of HIP. In my opinion, the problem is different: even if AMD supported CUDA proper, that still won't lead them anywhere because the reason people use Nvidia is not so much CUDA itself (although it has good tools) but Nvidia's proprietary libraries: cuBLAS, cuDNN, cuFFT etc. If AMD provided good implementations of similar libraries with OpenCL and shipped them with their drivers (similarily to Nvidia's CUDA toolkit) that would be much better than supporting CUDA, but leaving the ecosystem bare bones again...
    - gbrown 9y ago
      https://github.com/clMathLibraries https://github.com/clMathLibraries
      - dragandj 9y ago
        There are even better independent open-source equivalents for some of those, but that is not the point. Good luck compiling some of AMD's clMath libraries without issues. Then, you're "just" left with packaging them properly. Performance is also not stellar compared to Nvidia. Meanwhile, when you install Nvidia's toolkit, you're set with everything.
jonathanpoulter 9y ago
What impact does Google's Tensor Processing Unit have on the answer to this question?
- throwaway91111 9y ago
  Well, AFAIK, it isn't for sale; you have to go through their cloud offering. So it's not impactful at all on what GPU to get.
  - ktta 9y ago
    I mean it can have an effect on where the answer is to not get a GPU at all and use the cloud service.
- neurotech1 9y ago
  AFAIK AMD has not announced any Tensor Processing Unit or specific tensor acceleration capability. nVidia Volta [0] does have a Tensor Processing core optimization. https://www.nvidia.com/en-us/data-center/volta-gpu-architecture/ https://www.nvidia.com/en-us/data-center/volta-gpu-architect...
  - ktta 9y ago
    'Tensor processing' isn't some special kind of compute. The TPUs (which were badly named by google) are just specifically designed ASICs with lower precision which have much lower power consumption than GPUs, therefore cost effective for inference, which google does more than anyone on earth. 'Tensor Processing Unit' has become some what a phrase used to confuse (trick?) people into thinking it's some new type of processing, but its not. If the GPU says it has a Tensor Processing Unit, it just means it operates with a lower precision, but you have to remember that the overall power consumption of the GPU doesn't really change at full use compared to using higher precision chips. So you're actually missing out on the cost effectiveness of using lower precision. If anything the 'Tensor Processing Units' just take up unnecessary die space when included high higher precision units because they're bad for training compared to higher precision compute units.
tanilama 9y ago
My recommendation: 1080 Ti or 1080 GTX. There is no reason, not to get a Pascal GPU at this point, the performance is simply superior. For a lot of models, we are talking about days of training time, so 20%-30% time saving is significant, and we are not even touching the part that large VRAM enables bigger batch size which you won't get in middle/low end GPUs.
zeptomu 9y ago
There is an interesting offer by Hetzner that's available for some month now. They provide a dedicated server with a GTX 1080 for ~99e/month (111$/month) with adequate CPU (i7-6700), 64G Memory, 500G disk space and 50TB Traffic - there are also on-demand offerings by GCP and AWS, but I do not think they can match the offer by Hetzner: https://www.hetzner.de/us/hosting/produkte_rootserver/ex51ssd-gpu https://www.hetzner.de/us/hosting/produkte_rootserver/ex51ss.... Keep in mind that I am talking about R&D, in particular training of networks - which has lower expectations on availability than e.g. later inference. Disclosure: I am not affiliated with Hetzner and did not test it yet, but I have a ~40e dedicated machine there for occasional number crunching and everything worked so far (no availability or hardware issues). Furthermore I am not sure about the exact specifications (Memory-Size of GPU) or if there are different types of 1080s that differ significantly in Deep Learning Performance (see also: https://github.com/jcjohnson/cnn-benchmarks https://github.com/jcjohnson/cnn-benchmarks).
- throwaway32131 9y ago
  The electricity will likely cost a significant chunk of that, if one were running that full time in Germany. 0.5 * 24 * 30 * (30 c/kWH) = $108 However, it'd be a third of that in the states (may be cheaper in WA).
  - dahauns 9y ago
    Industrial consumers in Germany pay nowhere near 30c/kwh, more like 10-15c. (depending on size and exemptions)
- visionscaper 9y ago
  I might be mistaken, but these servers just have one GTX 1080.
ThePhysicist 9y ago
I recently bought an Alienware 13 R3 for deep learning (okay, and gaming...) and I'm quite satisfied so far. In addition to the built-in Titan 1060 with 6 GB of DDR5 it's possible to get an external graphics accelerator for additional performance as well (I haven't tried this yet though).
lunulata 9y ago
a link to a 500 error, way to trash up the ycombinator feed.
DrNuke 9y ago
GTX 1070 8GB has the best performance/cost at May 2017 and a good laptop costs as much as an equivalent desktop at below $2k.
lazylizard 9y ago
http://ambermd.org/gpus/benchmarks.htm http://ambermd.org/gpus/benchmarks.htm
KerrickStaley 9y ago
Question: if I'm learning about neural networks and want to e.g. train a network to recognize MNIST digits, do I need a discrete graphics card (probably attached to a VPS that I would rent)? Or can I use the i5 Kaby Lake (which has an integrated GPU) in my laptop to train my network?
- ktta 9y ago
  You actually don't even need the iGPU for simple MNIST training. The CPU is enough. Check out the sample MNIST training example with Tensorflow. If you are asking about using the iGPU for a bit more power, most frameworks don't support them. The only frameworks with which you can use with iGPUs are which have OpenCL support, like caffe[1] (even for those you need to have a recent enough CPU) Tensorflow either uses CPU or CUDA (Not even AMD, but the support is coming for openCL) [1]:https://github.com/BVLC/caffe/tree/opencl https://github.com/BVLC/caffe/tree/opencl
- daveguy 9y ago
  To train an NN to recognize MNIST digits you need any cpu made in the last 5 years. Last 10-20 years if you want to wait a little longer. MNIST recognition was first done by convolutional neural network in the late 90s. With less than 1% error. Here's a link to the paper: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf
- p1esk 9y ago
  MNIST is a toy problem. It shouldn't take you more than a few days or even hours to completely understand it. Do yourself a favor and get a GPU because soon you will want to try bigger networks on real world images. 1060, 1070, or 1080Ti are all good choices, depending on your budget and ambitions.
- solomatov 9y ago
  Simple network form tensroflow's MNIST tutorial trains in about 30 minutes on my macbook pro.
- marvy 9y ago
  As other people have said, for MNIST, CPU is fine. One core. You can get over 90% accuracy with a network with just 1 hidden layer with only a few seconds of training. Or something like that. If you've never played with MNIST before, it's kind of amazing how easy it is. For instance, the following idea "works", in that you get results that are pretty bad, but much better than chance. (Even more than 50% right, I think.) Suppose you have 10 arrays: zeros: each element of this array is a picture of a "0" ones: each element is a picture of a "1" ... nines: each element is a "9" Compute the average of each array: the "average" 0, the average 1, and so on. Then classify new digits based on which average element they are closest to, using Euclidean distance. This sounds way too dumb to do any good, but the MNIST digits are normalized so well that this actually does something. Even better: you can have a neural "network" that has zero hidden layers. This actually achieves almost respectable performance, believe it or not.
Sephr 9y ago
Here's a much more cost effective answer: Use https://cloud.google.com/tpu/ https://cloud.google.com/tpu/ Unless you have an unlimited supply of free electricity and don't care about the increased hardware management overhead, it's a waste of money to buy Pascal GPUs for large-scale deep learning. The following cards have much more optimized deep learning silicon and are publicly available /right now/: - Nvidia Tesla V100 (Tensor cores only: 120 TFLOPS FP16) - Google TPU2 (180 TFLOPS FP16) Additionally, Intel Xeon chips with the Nervana deep learning accelerator built-in will probably be available early next year. If you must control the physical hardware yourself and can't use cloud services, go buy Tesla V100s or wait for the Nervana Xeons.
- KaoruAoiShiho 9y ago
  That makes no sense as the TPU2 is not really out yet for actual consumers. AFAIK it's only in closed alpha right now, so if you're actually doing stuff right now it's not a real option. There's also no pricing so the "cost effective" remark can't really be seen.
  - netheril96 9y ago
    If and when it is for sale, I wonder how many problems with the hardware and drivers customers will face over the years.
locusm 9y ago
How hard does the GPU work on these tasks? Is it Bitcoin mining level power consumption?
- jacquesm 9y ago
  All out. This is typical: nvidia-smi Tue May 23 01:41:58 2017 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 375.39 Driver Version: 375.39 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Graphics Device Off | 0000:01:00.0 On | N/A | | 57% 84C P2 257W / 250W | 11010MiB / 11171MiB | 95% Default | +-------------------------------+----------------------+----------------------+ That's during training. I'm running a minimal desktop to have maximum ram for the minibatch (and it still isn't enough but it will have to do).
ktta 9y ago
If anyone here doesn't want to spend money on $500+ GPU (or the $1k+ ones!), then I'd suggest getting the lowest tier Nvidia GPU for $100~[1] If that GPU is a real bottleneck for you, then you're much better off spending money on GCP/AWS's GPU offerings. That's because consumer GPUs get superseeded every year and online offering's price will only go down. So you can spend 10% of $1k every year and keep getting better return on compute / dollar every year. [1]:https://www.newegg.com/Product/Product.aspx?Item=N82E16814487296 https://www.newegg.com/Product/Product.aspx?Item=N82E1681448...
- solomatov 9y ago
  >If that GPU is a real bottleneck for you, then you're much better off spending money on GCP/AWS's GPU offerings. That's because consumer GPUs get superseeded every year and online offering's price will only go down. GCP and AWS have old GPUs and they are really really expensive. If you expect to run workloads for a long time, it would be more cost efficient to buy your own hardware.
  - ktta 9y ago
    They are very, very few cases where getting an expensive GPU - anything more than a Nvidia 1060 would make financial sense. The 1050 is a beginner card is it is perfectly fine to learn and run small nets. More importantly, you can decide if machine learning is for you. Then, comes to second investment, which is actually running real-world models. Although the online GPU offering is expensive (you can also look around for cloud GPUs with lower SLA requirements for a lot cheaper) you'd be using them for a lot lesser time. Even if you go with the big three, you can get good pricing if you look around. AWS has a single GPU with 1,536 CUDA cores and 4GB RAM. Looking at the Spot Pricing[1] which is $0.25 an hour, you can get 3000 hours of compute for $750, which is the price of 1080Ti (which is the most cost effective card on the market today). Now, I would say, 3000 hours is more than enough time one needs to run whatever they want to. You can get a lot more if you go with some other service other than the big three. If your usage exceeds that, then you are probably using it for commercial purposes, in which case you need to account for a lot more variables (downtime, maintenance, etc). Then you should also consider the deprecating perf/$ your GPU gives you every year over the new ones in the market - especially since the perf jump is a lot more than what we're seeing with CPUs. If you are doing it for learning, I'd say you're doing something wrong, because you shouldn't need that much power. Also, consider the electricity costs and possible over usage of your computer. [1]:https://aws.amazon.com/ec2/spot/pricing/ https://aws.amazon.com/ec2/spot/pricing/ (check US East N. Virginia)
strin 9y ago
The link is not working for me now.
dkobran 9y ago
GTX drivers become crippled when the card detects the presence of a virtual environment. This means you can't run GTX in the cloud, otherwise, cloud GPU prices would be much lower. Without the availability of GTX, we've been trying just about everything at Paperspace to bring prices down and make the cloud a viable option for GPU compute. The argument being, there are real benefits to running in the cloud like on-demand scalability, lack of upkeep, minimal upfront costs, and of course, running a production application :) There are other indirect cost saving eg power consumption which can be quite significant when training models for long periods of time. Would love to see a total cost of ownership figure added to this post.
- deleted 9y ago
  [deleted]
- deleted 9y ago
  [deleted]
- deleted 9y ago
  [deleted]
- colejohnson66 9y ago
  Citation please for the crippled drivers?
  - dkobran 9y ago
    The citation is we're building a GPU cloud and have tested almost every GPU in existence :) Just kidding, here are a few examples: http://vfio.blogspot.com.au/2014/08/vfiovga-faq.html http://vfio.blogspot.com.au/2014/08/vfiovga-faq.html https://www.reddit.com/r/linux/comments/2twq7q/nvidia_apparently_pulls_some_shady_shit_to_do/ https://www.reddit.com/r/linux/comments/2twq7q/nvidia_appare... https://www.redhat.com/archives/libvirt-users/2014-October/msg00029.html https://www.redhat.com/archives/libvirt-users/2014-October/m... I just quickly googled this so there are probably better sources. Some of these are old but I can tell you firsthand that this is still the case. There are workarounds for certain hypervisors (KVM mainly) but it's very unlikely that this would be deployed in a production environment.
boulos 9y ago
Disclaimer: I work on Google Cloud. I saw a lot of "should we use Cloud, no its crazy a GPU only costs $X". The key is that if you believe GPUs are going to get updated every year, and/or the best thing for ML may change (see TPU and plenty of startups with custom hardware) then suddenly buying hardware for 24? 36? months isn't as obvious. We (and AWS and Microsoft) have K80s because Maxwell wasn't a sufficiently friendly all around part. We're all going to offer Pascal P100s and in the future V100s. The challenge for buying your own is that P100s are available now-ish and V100s may be available in less than 12 months. Buying a P100 or similar part today doesn't mean it won't still be working in a year, but it will suddenly mean you've bought a part that now has much worse !/$ in just N months. If you have an accounting team that is spreading your $Xk over those 36 months, the reality is that you have two options: tell everyone they have to make use the old parts ("we're not buying new GPUs until it's been 36 months!") or realize you're going to get a lot less use out of them. To be clear, the progress in this space is really impressive. And the same problem above applies to us (the cloud providers). Despite my obvious bias, if I were fired today, I'd be renting to do deep learning just based on the roadmap alone (not to mention the ability to suddenly spin up and down). Again, Disclosure: I work on Google Cloud and want to sell you things that train ML models :).
- brians 9y ago
  Is this any different than advising 18 month depreciation for GPUs?
- tensor 9y ago
  The problem is that a month of GPU time on a cluster can buy you the hardware itself. If you are doing serious deep learning work it's just not cost effective at this point. If the costs come down by half or more, it may start looking viable for people who need a lot of resources.
  - boulos 9y ago
    Can you explain your math? A K80 on GCE is $.7/hr x 730 => $511/month if you were really 24x7. A K80 (and really we sell them by the die not the board) is more than $1000. I don't disagree that a consumer board is about that price, but they're not apples to apples. (Either in memory size, reliability or both). I'm fine with that being the real complaint: (major) cloud providers only sell the Tesla class boards, and they're really expensive ;).
throwaway32131 9y ago
The trouble with getting Pascal now is that the card may end up depreciating very very sharply in a year, when Volta with its Tensor unit comes out.
fest 9y ago
I'm wondering about two things: 1. Can laptops with, say NVidia 1070 or 1080 GPUs keep them cool at their stock frequencies for a few hours? My work laptop with non-U i7 (Thinkpad T440p) starts thermal throttling in just a few minutes when I compile something large. 2. Wouldn't two NVidia 1060 or 1070 outperform a single 1080 for training, assuming batch size is kept low enough so that each batch fits in a single card's memory?
- mmusc 9y ago
  1. Depends on the laptop really. Some have better cooling than others. I'm training models for a few hours every day with an asus rog that has a 1060, and I'm not experiencing any throttling.
peeb 9y ago
Mirror: https://web.archive.org/web/20170522190506/http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/ https://web.archive.org/web/20170522190506/http://timdettmer...
sabalaba 9y ago
Short answer is that you can save weeks of work putting together a machine by just buying from Lambda (We of course use the 1080Tis): https://lambdal.com/deep-learning-devbox https://lambdal.com/deep-learning-devbox On average it takes a SWE or DL engineer a few days to set up a unit from scratch. Your company probably burns over $2,000/day so every day your DL engineer or SWE isn't up and running costs you money.

166 points by laktak 9y ago

Google cached version: http://webcache.googleusercontent.com/search?q=cache:04az_uBF27EJ:timdettmers.com/2017/04/09/which-gpu-for-deep-learning/ http://webcache.googleusercontent.com/search?q=cache:04az_uB...

godelski 9y ago
Thank you, I was getting Error 500 - Hug Of Death

ScottBurson 9y ago

Site is slammed, but I think this is from a couple years ago. -- Ah, it's been updated. Is there anything better now than the GTX 1080 Ti?

Analemma_ 9y ago
The Titan Xp is better, but it's not so much better that it's worth almost twice the price.
- paulsutter 9y ago
  Two 1080TIs are way better than one Titan Xp and cost about the same (2x$700 vs $1200). Each TI has 11GB of RAM vs the TitanX's 12GB, and each TI is nearly as fast. At the NVIDIA conference all the second tier hosting companies were promoting the P100 (at NVIDIAs insistence) but when pressured admitted that their big customers now deploy 1080TIs. Paying for P100s is sort of a clown move even if you're spending someone else's money. The P100 starts at like 10x the price of the 1080TI and isn't much faster and again has just 12GB of RAM (16GB if you pay $4000 more, ludicrous)
  - mattnewton 9y ago
    Usually, but the caveat is they are only better for workloads that are easily paralizable (like hyperparameter search). Multi-gpu models are still very complicated in my experience. And the Xp is a bit faster and has 1gb more memory, which is useful in some edge cases (like large or 3D covnets) Also if there do build with multiple gpus, watch out that you can give both cards a full 16 pci lanes, not a given in a lot of motherboards.
- sp332 9y ago
  Details https://arstechnica.com/gadgets/2017/04/nvidia-titan-xp-gp-102-details-price/ https://arstechnica.com/gadgets/2017/04/nvidia-titan-xp-gp-1... If your machine learning problem is sized to fit 12GB better than 11GB, it could save a lot of time. But unless you're really in a hurry, it's still probably not worth the huge price bump.

sgt101 9y ago

No mention of k80's k40's...

chronic940 9y ago
K80s and K40s have a K. That's two generations old. We are currently on P and about to be V. If you think K80s are good, you are far far behind the times in machine learning. A single 1080Ti even in a 4U server outperforms 2 K80s.
- sgt101 9y ago
  Ok - p100's? The thing is that a K80 costs.... £4k what is it that people are paying for?
  - dragandj 9y ago
    ECC memory (not that everyone really need that)...
    - sgt101 9y ago
      In a tensorflow job I find it hard to imagine that a memory error would make a jot of difference... wouldn't the error simply be removed by the network training? If the error was in the execution then I imagine that the epoch would fail and it would then just be a matter of restarting.
- arnon 9y ago
  If you're running a rackmount server, you need the Tesla series. We found that the GeForces tend to burn out when under heavy load, whereas we've not had a single Tesla series ever burn out.
  - maksimum 9y ago
    Ooh ooh tell us more. Which GeForces and which server chassis? Adequate power supply?

ericfrederich 9y ago

I'm guessing you don't... you use the cloud, right?

exelius 9y ago
If you're doing a lot of experimentation, it can make more sense to train your models locally (so you're not paying server / data transfer costs every time you test).
antognini 9y ago
For a hobbyist, it probably makes sense to just use some GPU instances on AWS. But it doesn't really take that long for it to become cost-effective to buy a GPU. It only takes a couple of months of full-time use before it becomes cost-effective to purchase the GPU. Even if you're building the entire machine from scratch it takes under a year. (At least, that was my estimate when I was building a rig earlier this year.) If you're doing any serious deep learning projects it's pretty easy to have a model training nearly 100% of the time.
- nightski 9y ago
  Not to mention, I can imagine it's frustrating uploading many gigs of data to amazon.
  - treehau5 9y ago
    extremely frustrating. Internet service is getting worse this in country, not better, sadly. The only lucky ones are those who live in competitive markets (lol) and those who had google fiber break some ground. The anti-competitiveness is getting worse too. Charter just took over TWC in my area.
  - zeptomu 9y ago
    It is not that bad. Either you've your training data local (unlikely) or it is already available in the "cloud" (i.e. a public-facing service). Let's say a typical training set (raw data) is 100G. Assuming 2.5MB/s Upload capacity with local data, you've uploaded it to your deep learning machine in half a day (100000 / 2.5 / 3600 ~ 11 hours) - which is not that much, as most of your time will be used for development and fine-tuning of your deep learning tool chain anyway. In most cases the data is accessible via a public-facing service, and assuming 1GBit/s bandwidth you've downloaded 100G in 13 minutes (100000 / 125 / 60 ~ 13 minutes).

jacquesm 9y ago

tldr: GTX1080ti They're fairly easy to get now too, in the beginning it was rather hard to get them.

astrodust 9y ago
You can get an 8GB GTX 1080 for 2/3rds the price and it offers 80% of the performance. If you don't need the 11GB of memory, it's a steal. If you're building a rig with multiple cards, 1080 might be an economical way to get great performance. If you're looking for maximal compute density, 1080 Ti is the way to go.
- ex3ndr 9y ago
  It will also require much more power and cooling this system will rise a cost and reduce stability of the system.
  - astrodust 9y ago
    The power consumption of a 1080 is actually lower than a 780 or 980, plus a 1080 Ti uses about 50W more power when pinned, so I'm not sure where you're coming from here.

AlphaSite 9y ago

Interestingly, AMD is claiming that their upcoming Vega Frontier Editions chips outperform Nvidia's current P100 Deep Learning chip. http://hexus.net/media/uploaded/2017/5/30f5633b-1bbf-49b7-9f31-7890e20db28f.jpg http://hexus.net/media/uploaded/2017/5/30f5633b-1bbf-49b7-9f...

sp332 9y ago
I think signs are pointing to Vega being a HPC beast. But what they really need is the software ecosystem and support, and so far that hasn't been there. So while the new hardware looks cool, I'm really waiting for an announcement that OpenCL tooling got a lot better, or that CUDA is getting first-party support from AMD, anything to tempt those customers away from nvidia.
- dragandj 9y ago
  CUDA is getting support from AMD in the form of HIP. In my opinion, the problem is different: even if AMD supported CUDA proper, that still won't lead them anywhere because the reason people use Nvidia is not so much CUDA itself (although it has good tools) but Nvidia's proprietary libraries: cuBLAS, cuDNN, cuFFT etc. If AMD provided good implementations of similar libraries with OpenCL and shipped them with their drivers (similarily to Nvidia's CUDA toolkit) that would be much better than supporting CUDA, but leaving the ecosystem bare bones again...
  - gbrown 9y ago
    https://github.com/clMathLibraries https://github.com/clMathLibraries
    - dragandj 9y ago
      There are even better independent open-source equivalents for some of those, but that is not the point. Good luck compiling some of AMD's clMath libraries without issues. Then, you're "just" left with packaging them properly. Performance is also not stellar compared to Nvidia. Meanwhile, when you install Nvidia's toolkit, you're set with everything.

jonathanpoulter 9y ago

What impact does Google's Tensor Processing Unit have on the answer to this question?

throwaway91111 9y ago
Well, AFAIK, it isn't for sale; you have to go through their cloud offering. So it's not impactful at all on what GPU to get.
- ktta 9y ago
  I mean it can have an effect on where the answer is to not get a GPU at all and use the cloud service.
neurotech1 9y ago
AFAIK AMD has not announced any Tensor Processing Unit or specific tensor acceleration capability. nVidia Volta [0] does have a Tensor Processing core optimization. https://www.nvidia.com/en-us/data-center/volta-gpu-architecture/ https://www.nvidia.com/en-us/data-center/volta-gpu-architect...
- ktta 9y ago
  'Tensor processing' isn't some special kind of compute. The TPUs (which were badly named by google) are just specifically designed ASICs with lower precision which have much lower power consumption than GPUs, therefore cost effective for inference, which google does more than anyone on earth. 'Tensor Processing Unit' has become some what a phrase used to confuse (trick?) people into thinking it's some new type of processing, but its not. If the GPU says it has a Tensor Processing Unit, it just means it operates with a lower precision, but you have to remember that the overall power consumption of the GPU doesn't really change at full use compared to using higher precision chips. So you're actually missing out on the cost effectiveness of using lower precision. If anything the 'Tensor Processing Units' just take up unnecessary die space when included high higher precision units because they're bad for training compared to higher precision compute units.

tanilama 9y ago

My recommendation: 1080 Ti or 1080 GTX. There is no reason, not to get a Pascal GPU at this point, the performance is simply superior. For a lot of models, we are talking about days of training time, so 20%-30% time saving is significant, and we are not even touching the part that large VRAM enables bigger batch size which you won't get in middle/low end GPUs.

zeptomu 9y ago

There is an interesting offer by Hetzner that's available for some month now. They provide a dedicated server with a GTX 1080 for ~99e/month (111$/month) with adequate CPU (i7-6700), 64G Memory, 500G disk space and 50TB Traffic - there are also on-demand offerings by GCP and AWS, but I do not think they can match the offer by Hetzner: https://www.hetzner.de/us/hosting/produkte_rootserver/ex51ssd-gpu https://www.hetzner.de/us/hosting/produkte_rootserver/ex51ss.... Keep in mind that I am talking about R&D, in particular training of networks - which has lower expectations on availability than e.g. later inference. Disclosure: I am not affiliated with Hetzner and did not test it yet, but I have a ~40e dedicated machine there for occasional number crunching and everything worked so far (no availability or hardware issues). Furthermore I am not sure about the exact specifications (Memory-Size of GPU) or if there are different types of 1080s that differ significantly in Deep Learning Performance (see also: https://github.com/jcjohnson/cnn-benchmarks https://github.com/jcjohnson/cnn-benchmarks).

throwaway32131 9y ago
The electricity will likely cost a significant chunk of that, if one were running that full time in Germany. 0.5 * 24 * 30 * (30 c/kWH) = $108 However, it'd be a third of that in the states (may be cheaper in WA).
- dahauns 9y ago
  Industrial consumers in Germany pay nowhere near 30c/kwh, more like 10-15c. (depending on size and exemptions)
visionscaper 9y ago
I might be mistaken, but these servers just have one GTX 1080.

ThePhysicist 9y ago

I recently bought an Alienware 13 R3 for deep learning (okay, and gaming...) and I'm quite satisfied so far. In addition to the built-in Titan 1060 with 6 GB of DDR5 it's possible to get an external graphics accelerator for additional performance as well (I haven't tried this yet though).

lunulata 9y ago

a link to a 500 error, way to trash up the ycombinator feed.

DrNuke 9y ago

GTX 1070 8GB has the best performance/cost at May 2017 and a good laptop costs as much as an equivalent desktop at below $2k.

lazylizard 9y ago

http://ambermd.org/gpus/benchmarks.htm http://ambermd.org/gpus/benchmarks.htm

KerrickStaley 9y ago

Question: if I'm learning about neural networks and want to e.g. train a network to recognize MNIST digits, do I need a discrete graphics card (probably attached to a VPS that I would rent)? Or can I use the i5 Kaby Lake (which has an integrated GPU) in my laptop to train my network?

ktta 9y ago
You actually don't even need the iGPU for simple MNIST training. The CPU is enough. Check out the sample MNIST training example with Tensorflow. If you are asking about using the iGPU for a bit more power, most frameworks don't support them. The only frameworks with which you can use with iGPUs are which have OpenCL support, like caffe[1] (even for those you need to have a recent enough CPU) Tensorflow either uses CPU or CUDA (Not even AMD, but the support is coming for openCL) [1]:https://github.com/BVLC/caffe/tree/opencl https://github.com/BVLC/caffe/tree/opencl
daveguy 9y ago
To train an NN to recognize MNIST digits you need any cpu made in the last 5 years. Last 10-20 years if you want to wait a little longer. MNIST recognition was first done by convolutional neural network in the late 90s. With less than 1% error. Here's a link to the paper: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf
p1esk 9y ago
MNIST is a toy problem. It shouldn't take you more than a few days or even hours to completely understand it. Do yourself a favor and get a GPU because soon you will want to try bigger networks on real world images. 1060, 1070, or 1080Ti are all good choices, depending on your budget and ambitions.
solomatov 9y ago
Simple network form tensroflow's MNIST tutorial trains in about 30 minutes on my macbook pro.
marvy 9y ago
As other people have said, for MNIST, CPU is fine. One core. You can get over 90% accuracy with a network with just 1 hidden layer with only a few seconds of training. Or something like that. If you've never played with MNIST before, it's kind of amazing how easy it is. For instance, the following idea "works", in that you get results that are pretty bad, but much better than chance. (Even more than 50% right, I think.) Suppose you have 10 arrays: zeros: each element of this array is a picture of a "0" ones: each element is a picture of a "1" ... nines: each element is a "9" Compute the average of each array: the "average" 0, the average 1, and so on. Then classify new digits based on which average element they are closest to, using Euclidean distance. This sounds way too dumb to do any good, but the MNIST digits are normalized so well that this actually does something. Even better: you can have a neural "network" that has zero hidden layers. This actually achieves almost respectable performance, believe it or not.

Sephr 9y ago

Here's a much more cost effective answer: Use https://cloud.google.com/tpu/ https://cloud.google.com/tpu/ Unless you have an unlimited supply of free electricity and don't care about the increased hardware management overhead, it's a waste of money to buy Pascal GPUs for large-scale deep learning. The following cards have much more optimized deep learning silicon and are publicly available /right now/: - Nvidia Tesla V100 (Tensor cores only: 120 TFLOPS FP16) - Google TPU2 (180 TFLOPS FP16) Additionally, Intel Xeon chips with the Nervana deep learning accelerator built-in will probably be available early next year. If you must control the physical hardware yourself and can't use cloud services, go buy Tesla V100s or wait for the Nervana Xeons.

KaoruAoiShiho 9y ago
That makes no sense as the TPU2 is not really out yet for actual consumers. AFAIK it's only in closed alpha right now, so if you're actually doing stuff right now it's not a real option. There's also no pricing so the "cost effective" remark can't really be seen.
- netheril96 9y ago
  If and when it is for sale, I wonder how many problems with the hardware and drivers customers will face over the years.

locusm 9y ago

How hard does the GPU work on these tasks? Is it Bitcoin mining level power consumption?

jacquesm 9y ago
All out. This is typical: nvidia-smi Tue May 23 01:41:58 2017 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 375.39 Driver Version: 375.39 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Graphics Device Off | 0000:01:00.0 On | N/A | | 57% 84C P2 257W / 250W | 11010MiB / 11171MiB | 95% Default | +-------------------------------+----------------------+----------------------+ That's during training. I'm running a minimal desktop to have maximum ram for the minibatch (and it still isn't enough but it will have to do).

ktta 9y ago

If anyone here doesn't want to spend money on $500+ GPU (or the $1k+ ones!), then I'd suggest getting the lowest tier Nvidia GPU for $100~[1] If that GPU is a real bottleneck for you, then you're much better off spending money on GCP/AWS's GPU offerings. That's because consumer GPUs get superseeded every year and online offering's price will only go down. So you can spend 10% of $1k every year and keep getting better return on compute / dollar every year. [1]:https://www.newegg.com/Product/Product.aspx?Item=N82E16814487296 https://www.newegg.com/Product/Product.aspx?Item=N82E1681448...

solomatov 9y ago
>If that GPU is a real bottleneck for you, then you're much better off spending money on GCP/AWS's GPU offerings. That's because consumer GPUs get superseeded every year and online offering's price will only go down. GCP and AWS have old GPUs and they are really really expensive. If you expect to run workloads for a long time, it would be more cost efficient to buy your own hardware.
- ktta 9y ago
  They are very, very few cases where getting an expensive GPU - anything more than a Nvidia 1060 would make financial sense. The 1050 is a beginner card is it is perfectly fine to learn and run small nets. More importantly, you can decide if machine learning is for you. Then, comes to second investment, which is actually running real-world models. Although the online GPU offering is expensive (you can also look around for cloud GPUs with lower SLA requirements for a lot cheaper) you'd be using them for a lot lesser time. Even if you go with the big three, you can get good pricing if you look around. AWS has a single GPU with 1,536 CUDA cores and 4GB RAM. Looking at the Spot Pricing[1] which is $0.25 an hour, you can get 3000 hours of compute for $750, which is the price of 1080Ti (which is the most cost effective card on the market today). Now, I would say, 3000 hours is more than enough time one needs to run whatever they want to. You can get a lot more if you go with some other service other than the big three. If your usage exceeds that, then you are probably using it for commercial purposes, in which case you need to account for a lot more variables (downtime, maintenance, etc). Then you should also consider the deprecating perf/$ your GPU gives you every year over the new ones in the market - especially since the perf jump is a lot more than what we're seeing with CPUs. If you are doing it for learning, I'd say you're doing something wrong, because you shouldn't need that much power. Also, consider the electricity costs and possible over usage of your computer. [1]:https://aws.amazon.com/ec2/spot/pricing/ https://aws.amazon.com/ec2/spot/pricing/ (check US East N. Virginia)

strin 9y ago

The link is not working for me now.

dkobran 9y ago

GTX drivers become crippled when the card detects the presence of a virtual environment. This means you can't run GTX in the cloud, otherwise, cloud GPU prices would be much lower. Without the availability of GTX, we've been trying just about everything at Paperspace to bring prices down and make the cloud a viable option for GPU compute. The argument being, there are real benefits to running in the cloud like on-demand scalability, lack of upkeep, minimal upfront costs, and of course, running a production application :) There are other indirect cost saving eg power consumption which can be quite significant when training models for long periods of time. Would love to see a total cost of ownership figure added to this post.

deleted 9y ago
[deleted]
deleted 9y ago
[deleted]
deleted 9y ago
[deleted]
colejohnson66 9y ago
Citation please for the crippled drivers?
- dkobran 9y ago
  The citation is we're building a GPU cloud and have tested almost every GPU in existence :) Just kidding, here are a few examples: http://vfio.blogspot.com.au/2014/08/vfiovga-faq.html http://vfio.blogspot.com.au/2014/08/vfiovga-faq.html https://www.reddit.com/r/linux/comments/2twq7q/nvidia_apparently_pulls_some_shady_shit_to_do/ https://www.reddit.com/r/linux/comments/2twq7q/nvidia_appare... https://www.redhat.com/archives/libvirt-users/2014-October/msg00029.html https://www.redhat.com/archives/libvirt-users/2014-October/m... I just quickly googled this so there are probably better sources. Some of these are old but I can tell you firsthand that this is still the case. There are workarounds for certain hypervisors (KVM mainly) but it's very unlikely that this would be deployed in a production environment.

boulos 9y ago

Disclaimer: I work on Google Cloud. I saw a lot of "should we use Cloud, no its crazy a GPU only costs $X". The key is that if you believe GPUs are going to get updated every year, and/or the best thing for ML may change (see TPU and plenty of startups with custom hardware) then suddenly buying hardware for 24? 36? months isn't as obvious. We (and AWS and Microsoft) have K80s because Maxwell wasn't a sufficiently friendly all around part. We're all going to offer Pascal P100s and in the future V100s. The challenge for buying your own is that P100s are available now-ish and V100s may be available in less than 12 months. Buying a P100 or similar part today doesn't mean it won't still be working in a year, but it will suddenly mean you've bought a part that now has much worse !/$ in just N months. If you have an accounting team that is spreading your $Xk over those 36 months, the reality is that you have two options: tell everyone they have to make use the old parts ("we're not buying new GPUs until it's been 36 months!") or realize you're going to get a lot less use out of them. To be clear, the progress in this space is really impressive. And the same problem above applies to us (the cloud providers). Despite my obvious bias, if I were fired today, I'd be renting to do deep learning just based on the roadmap alone (not to mention the ability to suddenly spin up and down). Again, Disclosure: I work on Google Cloud and want to sell you things that train ML models :).

brians 9y ago
Is this any different than advising 18 month depreciation for GPUs?
tensor 9y ago
The problem is that a month of GPU time on a cluster can buy you the hardware itself. If you are doing serious deep learning work it's just not cost effective at this point. If the costs come down by half or more, it may start looking viable for people who need a lot of resources.
- boulos 9y ago
  Can you explain your math? A K80 on GCE is $.7/hr x 730 => $511/month if you were really 24x7. A K80 (and really we sell them by the die not the board) is more than $1000. I don't disagree that a consumer board is about that price, but they're not apples to apples. (Either in memory size, reliability or both). I'm fine with that being the real complaint: (major) cloud providers only sell the Tesla class boards, and they're really expensive ;).

throwaway32131 9y ago

The trouble with getting Pascal now is that the card may end up depreciating very very sharply in a year, when Volta with its Tensor unit comes out.

fest 9y ago

I'm wondering about two things: 1. Can laptops with, say NVidia 1070 or 1080 GPUs keep them cool at their stock frequencies for a few hours? My work laptop with non-U i7 (Thinkpad T440p) starts thermal throttling in just a few minutes when I compile something large. 2. Wouldn't two NVidia 1060 or 1070 outperform a single 1080 for training, assuming batch size is kept low enough so that each batch fits in a single card's memory?

mmusc 9y ago
1. Depends on the laptop really. Some have better cooling than others. I'm training models for a few hours every day with an asus rog that has a 1060, and I'm not experiencing any throttling.

peeb 9y ago

Mirror: https://web.archive.org/web/20170522190506/http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/ https://web.archive.org/web/20170522190506/http://timdettmer...

sabalaba 9y ago

Short answer is that you can save weeks of work putting together a machine by just buying from Lambda (We of course use the 1080Tis): https://lambdal.com/deep-learning-devbox https://lambdal.com/deep-learning-devbox On average it takes a SWE or DL engineer a few days to set up a unit from scratch. Your company probably burns over $2,000/day so every day your DL engineer or SWE isn't up and running costs you money.