5 ms·
It has a few things going for it: 1) Enterprise-grade design, with features like ECC. I like ECC. 2) VRAM is basically the limiting factor on running LLMs. On
by frognumber 3y ago
It has a few things going for it:
1) Enterprise-grade design, with features like ECC. I like ECC.
2) VRAM is basically the limiting factor on running LLMs. On my GPUs, I look at cost versus VRAM. Fitting a model in one PC helps. Fitting a model on one GPU helps more.
3) It's just over half the power (250W versus 450W). The enterprise cards are clocked a little bit slower, which dramatically reduces power.
At least on the older Ampere cards, they also fit better. The consumer cards had giant cooling, which made sticking several in a computer more complex.
- aydyn 3y ago> The enterprise cards are clocked a little bit slower, which dramatically reduces power. But you can (and should) undervolt RTX 4090s if you care about power efficiency. Anyone buying one would know how to.
- stavros 3y agoDoes that get them to 250 W?
- syntaxing 3y agoNot OP, but you can limit the max wattage using nvidia-smi. Here's a good article with the instructions and how you would determine the best wattage. https://betterprogramming.pub/limiting-your-gpu-power-consumption-might-save-you-some-money-50084b305845 https://betterprogramming.pub/limiting-your-gpu-power-consum...
- stavros 3y agoThat's really interesting, thanks! I didn't realize you could do this though just an option.
- ksec 3y ago> I look at cost versus VRAM. Fitting a model in one PC helps. Fitting a model on one GPU helps more. Yes unfortunately I dont see 128GB VRAM coming any time soon even with 512bit GDDR Memory Interface.
- deleted 3y ago[deleted]
- frognumber 3y agoI would very, very gladly pay for a GPU which took normal DIMMs, and let me get up to 256GB. I can buy a 32GB DIMM for <$50. Eight of them would do it for $400. YES, I know the performance hit, but I'm not limited so much by performance as by which models fit. That's true of a lot of other people too. Only way I can see this happening is if Intel or AMD get serious about AI. This would seriously undercut NVidia's business.
- syntaxing 3y ago3090 also has NVlink. So in terms of the computer, 2x3090 with nvlink looks like one GPU.