8 ms·
The increasing TDP trend is going crazy for the top-tier consumer cards: 3090 - 350W 3090 Ti - 450W 4090 - 450W 5090 - 575W 3x3090 (1050W) is less than 2x5
by jbarrow 2y ago
The increasing TDP trend is going crazy for the top-tier consumer cards:
3090 - 350W
3090 Ti - 450W
4090 - 450W
5090 - 575W
3x3090 (1050W) is less than 2x5090 (1150W), plus you get 72GB of VRAM instead of 64GB, if you can find a motherboard that supports 3 massive cards or good enough risers (apparently near impossible?).
- holoduke 2y agoCan you actually use multiple videocards easily with existing AI model tools?
- jbarrow 2y agoYes, though how you do it depends on what you're doing. I do a lot of training of encoders, multimodal, and vision models, which are typically small enough to fit on a single GPU; multiple GPUs enables data parallelism, where the data is spread to an independent copy of each model. Occasionally fine-tuning large models and need to use model-parallelism, where the model is split across GPUs. This is also necessary for inference of the really big models, as well. But most tooling for training/inference of all kinds of models supports using multiple cards pretty easily.
- benob 2y agoYes, multi-GPU on the same machine is pretty straightforward. For example ollama uses all GPUs out of the box. If you are into training, the huggingface ecosystem supports it and you can always go the manual route to put tensors on their own GPUs with toolkits like pytorch.
- qingcharles 2y agoYes. Depends what software you're using. Some will use more than one (e.g. llama.cpp), some commercial software won't bother.
- dpeterson 2y agoI just made a video on this very thing: https://youtu.be/JtbyA94gffc https://youtu.be/JtbyA94gffc
- iandanforth 2y agoSounds like you might be more the target for the $3k 128GB DIGITS machine.
- jbarrow 2y agoI’m really curious what training is going to be like on it, though. If it’s good, then absolutely! :) But it seems more aimed at inference from what I’ve read?
- bmenrigh 2y agoI was wondering the same thing. Training is much more memory-intensive so the usual low memory of consumer GPUs is a big issue. But with 128GB of unified memory the Digits machine seems promising. I bet there are some other limitations that make training not viable on it.
- tpm 2y agoIt will only have 1/40 performance of BH200, so really not enough for training.
- jbarrow 2y agoPrimarily concerned about the memory bandwidth for training. Though I think I've been able to max out my M2 when using the MacBook's integrated memory with MLX, so maybe that won't be an issue.
- ryao 2y agoTraining is compute bound, not memory bandwidth bound. That is how Cerebras is able to do training with external DRAM that only has 150GB/sec memory bandwidth.
- jdietrich 2y agoThe architectures really aren't comparable. The Cerebras WSE has fairly low DRAM bandwidth, but it has a huge amount of on-die SRAM. https://www.hc34.hotchips.org/assets/program/conference/day2/Machine%20Learning/HC2022_Cerebras_Final_v02.pdf https://www.hc34.hotchips.org/assets/program/conference/day2...
- cogman10 2y agoWhat I really don't like about it is low power GPUs appear to be a thing of the past essentially. An APU is the closest you'll come to that which is really somewhat unfortunate as the thermal budget for an APU is much tighter than it has to be for a GPU. There is no 75W modern GPU on the market.
- justincormack 2y agothe closest is the L4 https://www.nvidia.com/en-us/data-center/l4/ https://www.nvidia.com/en-us/data-center/l4/ but its a bit weird.
- moondev 2y agoRTX A4000 has an actual display output
- moondev 2y agoInnodisk EGPV-1101
- Scene_Cast2 2y agoI heavily power limited my 4090. Works great.
- winwang 2y agoYep. I use ~80% and barely see any perf degradation. I use 270W for my 3090 (out of 350W+).
- mikae1 2y agoPerformance per watt[1] makes more sense than raw power for most consumer computation tasks today. Would really like to see more focus on energy efficiency going forward. [1] https://en.wikipedia.org/wiki/Performance_per_watt https://en.wikipedia.org/wiki/Performance_per_watt
- epolanski 2y agoThat's s blind way to look at that imho. Doesn't work on me for sure. More energy means more power consumption, more heat in my room, you can't escape thermodynamics. I have a small home office, it's 6 square meters, during summer energy draw in my room makes a gigantic difference in temperature. I have no intention of drawing more than a total 400w top while gaming and I prefer compromising on lowering settings. Energy consumption can't keep increasing over and over forever. I can even understand it on flagships, they meant for enthusiasts, but all the tiers have been ballooning in energy consumption.
- bb88 2y agoIncreasing performance per watt means that you can get more performance using the same power. It also means you can budget more power for even better performance if you need it. In the US the limiting factor is the 15A/20A circuits which will give you at most 2000W. So if the performance is double but it uses only 30% more power, that seems like a worthwhile tradeoff. But at some point, that ends when you hit a max power that prevents people from running a 200W CPU and other appliances on the same circuit without tripping a breaker.
- epolanski 2y ago> Increasing performance per watt means that you can get more performance using the same power. I'm currently running a 150 watt GPU, and the 5070 has a 250 TDP. You are correct. I could get a 5070 and down volt it to work in 150ish range e.g. and get almost the same performance (at least not significantly different to notice in game). But I think you're missing the wider point of my complain: it's been from Maxwell that Nvidia hasn't produced major updates on the power consumption side of their architecture. Simply making bigger and denser chips on better nodes while keeping to increase the power draw and slapping DLSS4 is not really an evolution, it's laziness and milking the users. On top of that: the performance benefits we're talking about are really using DLSS4, which is artificially limited to the latest gen. I don't expect raw performance of this gen to exceed a 20% bump to the previous one when DLSS is off.
- marricks 2y agoI got into desktop gaming at the 970 and the common wisdom (to me at least, maybe I was silly) was I could get away with a lower wattage power supply and use it in future generations cause everything would keep getting more efficient. Hah...
- epolanski 2y agoYeah, do like me, I lower settings from "ultra hardcore" to "high" and keep living fine on a 3060 at 1440p for another few gens. I'm not buying GPUs that expensive nor energy consuming, no chance. In any case I think Maxwell/Pascal efficiency won't be seen anymore, with those RT cores you get more energy draw, can't get around that.
- mikepurvis 2y agoI feel similarly; I just picked up a second hand 6600 XT (similar performance to 3060) and I feel like it would be a while before I'd be tempted to upgrade, and certainly not for $500+, much less thousands.
- brokenmachine 2y ago8Gb VRAM isn't enough for newer games though.
- alyandon 2y agoI'm generally a 1080p@60hz gamer and my 3060 Ti is overpowered for a lot of the games I play. However, there are an increasing number of titles being released over the past couple of years where even on medium settings the card struggles to keep a consistent 60 fps frame rate. I've wanted to upgrade but overall I'm more concerned about power consumption than raw total performance and each successive generation of GPUs from nVidia seems to be going the wrong direction.
- epolanski 2y agoI think you can get a 5060 and simply down volt it some, you'll get more or less the same performance while reducing power draw sensibly.
- elorant 2y agoYou don't need to run them in x16 mode though. For inference even half that is good enough.
- ashleyn 2y agomost household circuits can only support 15-20 amps at the plug. there will be an upper limit to this and i suspect this is nvidia compromising on TDP in the short term to move faster on compute
- SequoiaHope 2y agoI wonder if they will start putting lithium batteries in desktops so they can draw higher peak power.
- jbarrow 2y agoThere's a company doing that for stovetops, which I found really interesting (https://www.impulselabs.com https://www.impulselabs.com)! Unfortunately, when training on a desktop it's _relatively_ continuous power draw, and can go on for days. :/
- SequoiaHope 2y agoYeah that stove is what I was thinking of! And good point on training. I don't know what use cases would be supported by a battery, but there's a marketable one I am sure we will hear about it.
- ryao 2y agoThey already use capacitors for that.
- SequoiaHope 2y agoBatteries and capacitors would serve different functions. Capacitors primarily isolate each individual chip and subsystem on a PCB from high frequency power fluctuations when digital circuits switch or larger loads turn on or off. You would still need to use capacitors for that. The purpose of the batteries would be to support high loads on the order of minutes that exceed the actual wall plug capacity to deliver electricity. I am thinking specifically of the stove linked in your sibling comment, which uses lithium batteries to provide sustained bursts of power to boil a pot of water in tens of seconds without exceeding the power ratings of the wall plug.
- saomcomrad56 2y agoIt's good to know can all heat our bedrooms while mining shitcoins.
- 6SixTy 2y agoNvidia wants you to buy their datacenter or professional cards for AI. Those often come with better perf/W targets, more VRAM, and better form factors allowing for a higher compute density. For consumers, they do not care. PCIe Gen 4 dictates a tighter tolerance on signalling to achieve a faster bus speed, and it took quite a good amount of time for good quality Gen 4 risers to come to market. I have zero doubt in my mind that Gen 5 steps that up even further making the product design just that much harder.
- throwaway48476 2y agoIn the server space there is gen 5 cabling but not gen 5 risers.
- throwaway2037 2y ago> gen 5 cabling Do you mean OCuLink? Honestly, I never thought about how 1U+ rackmount servers handle PCIe Gen5 wiring/timing issues between NVMe drives (front), GPUs/NICs (rear), and CPUs (middle).
- throwaway48476 2y agoOCuLink has been superseded by MCIO. I was speaking of the custom gen 5 cabled nvme backplane most servers have.
- dabinat 2y agoThis is the #1 reason why I haven’t upgraded my 2080 Ti. Using my laser printer while my computer is on (even if it’s idle) already makes my UPS freak out. But NVIDIA is claiming that the 5070 is equivalent to the 4090, so maybe they’re expecting you to wait a generation and get the lower card if you care about TDP? Although I suspect that equivalence only applies to gaming; probably for ML you’d still need the higher-tier card.
- iwontberude 2y agoThat’s because you have a Brother laser printer which charges its capacitors in the least graceful way possible.
- throwaway81348 2y agoPlease expand, I am intrigued!
- lukevp 2y agoThis happens with my Samsung laser printer too, is it not all laser printers?
- bob1029 2y agoIt's mostly the fuser that is sucking down all the power. In some models, it will flip on and off very quickly to provide a fast warm up (low thermal mass). You can often observe the impact of this in the lights flickering.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- dogline 2y agoIf my Brother laser printer starts while I have the ceiling fan going on the same circuit, the breaker will trip. That's the only thing in my house that will do it. It must be a huge momentary current draw.
- zitterbewegung 2y agoInstead of risers just use pcie ender cords and you can get 4x 3090's working with a creator motherboard (google one that you know can handle 4). You could also use a mining case to do the same. But, the advantage is that you can load a much more complex model easily (24GB vs 32GB is much easier since 24GB is just barely around 70B parameters).
- Geee 2y agoYeah, that's bullshit. I have a 3090 and I never want to use it at max power when gaming, because it becomes a loud space heater. I don't know what to do with 575W of heat.
- ryao 2y agoI wonder how many generations it will take until Nvidia launches a graphics card that needs 1kW.
- faebi 2y agoI wish mining was still a thing, it was awesome to have free heating in the cold winter.
- Arkhadia 2y agoIs it not? (Serious question)
- abrookewood 2y agoProbably not on GPUs - think it all moved to ASICs years ago.
- ryao 2y agoMining on GPUs was never very profitable unless you held the mined coins for years. I suspect it still is profitable if you are in a position to do that, but the entire endeavor seems extremely risky since the valuation increases are not guaranteed.
- SunlitCat 2y agoWhich didn't stop people gobbling up every available gpu in the late 2010's. (Which, in my opinion, was a contributing factor why VR pc gaming didn't take of when better VR headsets arrived just around that point.)
- echoangle 2y ago> Mining on GPUs was never very profitable unless you held the mined coins for years. If mining is only profitable after holding, it wasn't profitable. Because then you could have spent less money to just buy the coins instead of mining them yourself, and held them afterwards.
- wkat4242 2y agoYes but the memory bandwidth of the 5090 is insanely high
- porphyra 2y agosoon you'll need to plug your PC into the 240 V dryer outlet lmao (with the suggested 1000 W PSU for the current gen, it's quite conceivable that at this rate of increase soon we'll run into the maximum of around 1600 W from a typical 110 V outlet on a 15 A circuit)
- jmward01 2y agoYeah. I've been looking at changing out my home lab GPU but I want low power and high ram. NVIDIA hasn't been catering to that at all. The new AMD APUs, if they can get their software stack to work right, would be perfect. 55w TDP and access to nearly 128GB, admittedly at 1/5 the mem bandwidth (which likely means 1/5 the real performance for tasks I am looking at but at 55w and being able to load 128g....)
- skocznymroczny 2y agoIn theory yes, but it also depends on the workload. RTX 4090 is ranking quite well on the power/performance scale. I'd rather have my card take 400W for 10 minutes to finish the job than take only 200W for 30 minutes.
- abrookewood 2y agoSooo much heat .... I'm running a 3080 and playing anything demanding warms my room noticeably.