11 ms·
Benchmarking TensorFlow on Nvidia GeForce RTX 3090
- valine 6y agoSeems like a good speedup relative to the Titan, especially for the money. I’d be interested to see the performance relative to the 3080 though. There are obviously vram limitations with the 3080 but it would still be interesting to see the difference in raw compute performance. In games the 3090 only gives a 15% performance bump relative to the 3080. If that pattern holds for machine learning tasks there is probably a scenario where it makes sense to buy two 3080s rather than one 3090. If you are vram constrained then obviously the 3090 the way to go.
- hhhhhuu 6y agoA 3080 with 20gb is planned already
- YetAnotherNick 6y agoVery likely not with the same price as 3080. I am guessing it won't be much cheaper than 3090 as vram is expensive.
- wombatmobile 6y agoIf this isn't OT... Could you kindly advise what kind of computer would make sense to purchase to begin learning about ML? I was assuming I'd get a 3080. Should I get a case that could potentially house 2 x 3080's? Does the case require any special cooling considerations, or just whatever will fit the cards? What CPU would you get?
- gbrown 6y agoI'd honestly start with cloud options if learning is the only reason you're building the computer. You don't want to dump a bunch of money into depreciating GPUs if you're not going to end up using them. GPUs are only really required in ML if you want to do deep neural network stuff. You can do plenty in CPU on reasonable data sets using any modern laptop.
- valine 6y agoWell to start off I’m not advising buying two 3080s. I haven’t seen the benchmarks, and on top of that the 3080 doesn’t support SLI so if you do buy two of them you will need to be using software which can utilize two independent GPUs. If you’re just wanting to learn machine learning you don’t need anything particularly special. I think you would be happy with GTX 1070. There is also the cloud computing route where you basically rent the gpu from AWS. That will be initially more cost effective than buying your own hardware. One thing to keep in mind if you do go with the 3080 is the power consumption. Ampere cards are going to be much more power hungry than previous generations, and you will need to budget about 320W just to the graphics card. The recommended power supply for the 3080 is 850W.
- posix_compliant 6y agoNvidia recommends 750W for the power supply.
- secabeen 6y agoThat is true, but it all depends on how much the rest of your system uses. If you just have a mid-tier CPU and SSD, you may find that the rest of your system only uses 200W, and you can fit a 3080 on a 600W power supply. There are multiple reports of high-quality 600W PSUs working, so YMMV.
- leeter 6y agoIt's worth noting that this recommendation has nothing to do with delivered power.... a 500W can handle most systems on delivered power. This has to do with transients, the 3000 series cards have some massive transients that can easily trip OCP protection on powersupplies not designed for that kind of transient. A 700W powersupply is able to handle those transients much better than a 500W PSU is.
- paol 6y agoIf you're just beginning a 3080 is overkill, never mind two. Get a used 1080ti (or even 2080ti if not overpriced), it'll be cheaper and even has a bit more RAM.
- gambiting 6y agoIf you're "learning about ML" there is no point in buying anything. Just get the cloud compute instead, and for home use and testing literally anything will do. I have friends who work with ML professionally and even they say it's just hard to justify running any computations at home once you factor in the electricity and hardware cost - GCP compute just beats the cost, easily.
- claudeganon 6y agoAgreed. If you’re just learning or building hobby stuff, you can use Colab, Paperspace, or any number of other services for free or very cheaply.
- emteycz 6y agoI want a gaming computer than won't limit my future ML learning. Are there any suggestions for that use case?
- cauthon 6y agoBuy whatever makes you feel happy. I agree with gambiting that anything you choose won’t limit you
- gambiting 6y agoThere's no such thing as long as you buy an actual mid to high tier GPU. Even an ancient GTX1070 would be more than enough - and for sufficiently large datasets even an RTX3090 will take hours to process whatever you're crunching. Just buy a PC that you like for gaming(with an Nvidia gpu) and don't worry about ML yet - it's incredibly unlikely that you can pick something that would limit you in any way. Small datasets will run on anything, large datasets will take hours to process no matter what you run them on. It's not a "limit".
- claudeganon 6y agoSome off-the-shelf gaming PCs are not very Linux friendly though, so they should watch out for that, especially the laptop varieties. Getting a lot of the ML stuff working locally in Windows is a nightmare.
- l33tman 6y agoGo for it! Get a motherboard with reinforced PCIe slots for both GPUs though, the cheaper mobos only have one armoured slot. Also, you really should use the 3080 Founders Edition I guess for a dual setup as they blow out part of their heat in the back. Otherwise you need good air control and as thin 3080 cards as you can find (so there's some space between them in the case). Still waiting for good thermal benchmarks on this kind of setup... Of course if money is an issue, you are well off with only a single 3080 :) There is no point in buying the 20xx series anymore. The 30xx are twice as good for the same money (if you can get one).
- duhi88 6y agoWoah, there. They said they were just learning. No need to purchase special hardware until you're trying to run state of the art models. You can get very far on any laptop before hardware becomes the main blocker. And before building an ML machine, there are cloud compute options available for far cheaper.
- bitL 6y ago3080 has no NVLink, so 2x 3080 would communicate only via PCIe and it's unlikely they would form a single 20-40GB virtual RAM like what 2x2080Ti with NVLink did under Linux.
- kevingadd 6y agoI think for high throughput scenarios the 3090 probably has more headroom due to its higher TDP and better (larger) cooling solution, which might really matter here if you're driving the tensor cores at max the whole time.
- wnevets 6y agoMost video games probably aren't going to make the most of all of the extra CUDA cores on the 3090. I'm assuming that helps alot with machine learning, can someone who knows for sure confirm?
- Jhsto 6y agoMost parallel processing scales linearly with core count. But the 3090 is more interesting for machine learning because of its RAM, which it has 24GB against 3080's 10GB. With machine learning, you spent most of the time copying memory between the CPU and GPU, so being able to fit more data to it reduces computation latency.
- option 6y ago“ With machine learning, you spent most of the time copying memory between the CPU and GPU” - this is a sign that you are most likely doing it wrong. Yes, some operations are inherently bandwidth bound, but most important ones such as larger matrix multiplies (transformers) and convolutions are compute bound.
- Jhsto 6y agoSure, but compared to the 3080 I'd say that the main deal is the bigger RAM for copying reason than the increased core count.
- option 6y agoTIP: if you need to emulate a bigger batch with less RAM available - use gradient accumulation trick. Super easy to implement in Pytorch and it is already implemented as a single flag (accumulate_grad_batches) in Pytorch Lightning.
- p1esk 6y agoYour gradient accumulation trick involves multiple cpu to gpu transfers, which is precisely what the parent is trying to avoid with fitting a larger batch in gpu memory.
- p1esk 6y agoThere will be 20GB version of 3080 soon.
- oivey 6y agoThere’s been rumors of this, but it’s not confirmed. A big percentage of the cost of the 3090 (and 3080) is the GDDR6X. A 3080 with 20GB of GDDR6X will still be really expensive, so it’s unclear to me that they will actually release something like that. Potentially they could put that much RAM on a card and then use slower GDDR6, but that’s kind of an odd part in Nvidia’s product offerings because then the 3080 with more RAM would be slower in a lot of situations.
- bitL 6y agoThere are rumors about 48GB 3090 with older GDDR6 chips so having the same config for 3080 wouldn't be unexpected, at least until Micron could produce 2GB GDDR6X chips.
- duhi88 6y agowhat would be the benefit there? 20GB can't be that much cheaper than 24GB, right?
- nullifidian 6y ago2x 20gb 3080 for $1000 each is more cost effective than 2x 3090 for $1500 each, if your model fits into 20 gb.
- p1esk 6y agoMy guess is it’s going to be $999
- Sohcahtoa82 6y agoIt will be for the gamers who think that 10 GB isn't enough VRAM, and as a way for nVidia to have an answer for rumors that AMD's next GPU will have 16 GB.
- jakear 6y agoThat second table is aa good example of why always including units (or even just a "higher is better") is a good idea... I have no clue what I'm looking at. Edit: It's been edited, thx Evolution :) (or I totally glossed over it the first time around... but I don't think so)
- zamadatix 6y ago"Training performance in images processed per second"?
- mamon 6y agoDo you mean this table which has a caption over it that reads "Training performance in images processed per second" ?? Looks pretty self-explanatory.
- Sayrus 6y agoPerhaps it has been edited. Now the table contains the following title "Training performance in images processed per second". A "Higher is better" might still be interesting although redundant.
- hughes 6y agoEven after being edited, it's still wrong. It shows the significantly lower Inception4 performance as a "40% speedup" instead of 40% of baseline images/sec.
- TomVDB 6y agoEven after the edit, it's a bit confusing in that, without looking at the table, you get the impression that FP16 is slower than FP32.
- BookPage 6y agoHonestly I had an RTX Titan for home use for a while. Eventually I moved to just using a 2080 Super and it performed at nearly the same power for my models. If you don't need ALL the extra memory and have the space for a triple slot then the better value proposition by far for last gen seemed to be a good super.
- danbr 6y agoI just want to know how they installed the new nvidia cuda drivers without borking their Ubuntu/tf install.
- rkwasny 6y agoHi, we just installed everything using nvidia repo and .deb packages. + tf-nightly and other python libraries installed through pipenv
- paol 6y agoNvidia has official ppa's with all versions of CUDA, libcudnn and drivers. If you install from there you will not have problems. It helps keeping to Ubuntu LTS versions though, that's what they support best.
- opless 6y agoNVidia drivers b0rked rebooting my box for a long time. A couple of months ago, I removed all the references in apt sources, and followed the newer instructions (several times to get the right driver/cuda/tensorflow match) and my reboots are great, and only one GPU lock up so far (probably due to overheating - I've had to replace a couple of components flag as failed due to the heatwave in summer) Jupyter hub is just great, I'd like to implement better diagnostics though ... have yet to find a good tutorial for that as yet.
- bryan0 6y agoCan someone explain the difference between fp16 and fp32 in these benchmarks because the difference is pretty dramatic. I assume it's floating point precision(?) but why would lower precision be slower relatively on the 3090? For training jobs how does the precision impact accuracy of the model? Edit: clarified that I am referring to slower relative performance
- smallnamespace 6y agoFP16 is faster (units are images per second)
- bryan0 6y agoSorry I was referring to the relative performance, I edited my question to be clearer
- riku_iki 6y ago> FP16 is faster (units are images per second) But does model get quality hit: need to train for more steps before converging to the similar performance and have more parameters? FM16 obviously contains less information than FP32.
- bitL 6y ago3090 opted for bundling 2x FP32 units Bulldozer-style and now FP16 is processed by those cores as well, so FP16 and FP32 have the same performance (35.58TFlops). https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622 https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622
- bufo 6y agoNvidia nerfed at the software level the FP16 performance to disincentivize people from using this card as a TITAN / datacenter ML card replacement.
- my123 6y agoIt isn't at the software level, FP16 goes through the tensor cores on Turing onwards: https://www.anandtech.com/show/13973/nvidia-gtx-1660-ti-review-feat-evga-xc-gaming/2 https://www.anandtech.com/show/13973/nvidia-gtx-1660-ti-revi...)
- fareesh 6y agoIf I have a really remote location and I need to do on-premises inference, am I better off buying one of the gaming GPUs or are they far behind the T4, etc.?
- paol 6y agoIt's the opposite, the T4 is far behind gaming GPUs. Reasons to buy a T4: - You need the 16Gb RAM. You usually won't for inference, though. - You need single slot cards to use in server chassis. Even here you are likely better off buying a Quadros than a T4 - You need the high power efficiency of the T4 (75W TDP)
- fareesh 6y agoOh interesting. Is there a benchmark chart or something like that where I can review the performance of something like YOLOv3 across cards?
- liuliu 6y agoSeems to be good speedup overall relative to 2080 Ti (including FP16: see relatives 2080 Ti v Titan: https://lambdalabs.com/blog/2080-ti-deep-learning-benchmarks/ https://lambdalabs.com/blog/2080-ti-deep-learning-benchmarks...). This suggests we should see another Titan card that is even more expensive in the pipeline given the FP16 performance? Or maybe TF32 performance is going to be what NVIDIA promotes in this generation (only if they have better number than FP16?)?
- jjoonathan 6y agoHere's hoping for an A100 titan with un-nerfed FP64. The 3090 is twice as nerfed as previous generations, which were also bad at 1:32. Now it's 1:64 :(
- p1esk 6y agoFP64 is not needed for deep learning.
- jjoonathan 6y agoWhat makes you think I'm doing deep learning?
- p1esk 6y agoYou’re commenting on the post called “Benchmarking deep learning workloads”.
- deleted 6y ago[deleted]
- Dylan16807 6y agoThe FP64 units are a separate addition that eat a lot of die space, right? I wouldn't use the word "nerf" for the tradeoff between having more SMs versus having more features in the SMs.
- systemvoltage 6y agoAside: Nit: Don't use gradients for discrete categories in a graph. Use a discrete color palette that perceptually distances colors as much as possible using a tool like this: https://medialab.github.io/iwanthue/ https://medialab.github.io/iwanthue/
- motorcitycobra 6y agoI thought I read Nvidia was nerfing the GeForce cards. Does this disprove it?
- bitL 6y agoNo, 3090 has nerfed tensor cores and in some apps Titan RTX is 5x faster (Siemens NX). FP32 accumulate is at 0.5x like with 2080Ti, while Titan's is at 1x.
- fomine3 6y agoNVIDIA nerfs FP64 performance on consumer GeForce for recent years. It's critical for scientific calculations but not needed for ML. Alternatively they banned to run GeForce on datacenter.
- arijun 6y agoSee also Tim Detter's fantastic post on GPU performance (which doesn't use benchmarks for the latest cards but instead calculates performance with a model): https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/ https://timdettmers.com/2020/09/07/which-gpu-for-deep-learni... HN Discussion: https://news.ycombinator.com/item?id=24400603 https://news.ycombinator.com/item?id=24400603
- ml_hardware 6y agoThis is a poor comparison of performance. All of these networks are CNNs, and very old architectures at that. They are all probably memory bottlenecked which is why you see the consistent 50% improvement in FP32 perf. It is also not clear what batch sizes are being used for any of the tests. If you switch to FP16 training, you must increase the batch size to properly utilize the Tensor Cores. If you compare these cards at FP16 performance on large language models (think GPT-style with large model dimension), I am confident you will see Titan RTX outperform the 3090. The former has 130 TF/s of FP16.32 tensor core performance while the latter has only 70 TF/s. Link: https://www.nvidia.com/content/dam/en-zz/Solutions/geforce/ampere/pdf/NVIDIA-ampere-GA102-GPU-Architecture-Whitepaper-V1.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/geforce/a...
- binarymax 6y agoThe 3090 RTX is also $1000 cheaper than the Titan, so there's that. It would be nice if there was a good way to express value per dollar. Perhaps in GLUE accuracy and training time.
- ml_hardware 6y agoTotally agree! I think 3090 could be a lot more cost effective for researchers to dabble with NLP. But it really grinds my gears when people post these misleading benchmarks... the 3090 is handicapped at half-rate tensor core performance while the Titan RTX is not. So if you're someone who does their work mainly in FP32, you will see improved performance with the 3090. On the other hand, if you are an FP16 speed demon who needs to train GPT-3 over the weekend, stick with your Titans :)
- bitL 6y agoWhat do you think about TF32 in 3090? Could it replace FP32 with 5x speedup?
- ml_hardware 6y agoI've done a lot of work in ML numerics, and I think TF32 is a completely safe drop-in for FP32 for ML workloads. NVIDIA seems to think so too, which is why on A100 it won't even be an option, it will be the default mode for any FP32 matrix multiplies. But on 3090, I don't think the speedup will be 5x, it should be closer to like 2x. The 3090 has 35.6 TF/s at TF32 and the Titan RTX has 16.3 TF/s at FP32. Once again I think there is handicapping going on for 3090.
- deleted 6y ago[deleted]