3 ms·
Their GPUs have been very performant for my research (DNN training). However, their VRAM could be much larger. In my mind CPU ram is very cheap (for 32GB, it is
by ayakang31415 3y ago
Their GPUs have been very performant for my research (DNN training). However, their VRAM could be much larger. In my mind CPU ram is very cheap (for 32GB, it is currently around $65). But their GPUs for DNN that I use oh HPC are always less than 32GB but the GPUs are very pricey. Does anyone know why they don't increase their VRAM capacity so I can test models that require higher VRAM? Is VRAM considerably more expensive to attach to GPU versus CPU ram?
- samspenc 3y agoFolks on Reddit's /r/pcmasterrace have been discussing this for years - the consensus seems to be that Nvidia could add more VRAM to its GPUs without too much additional cost - but they don't want to, in order to push higher spending businesses and consumers to buy their more expensive chips to get more VRAM at exponential higher costs.
- AnotherGoodName 3y agoYeah their high VRAM cards (80gb h100) cost $40k. That's their price differentiator. It's the reason for their profits. nVidia is now <20% gaming in terms of revenue and that gaming revenue has much tighter margins than the 80% datacenter/professional market revenue which is almost all profit sine 80gb ram doesn't cost $40k more. So yes it'd be nice if nVidia made it so there was no reason for their $40k AI cards to exist. But they aren't going to do that.
- jules22 3y agoI hope Intel, as the underdog, can do mid-tier cards, but with lot more VRAM. I can see myself using multiple LLM models at once, not necessarily running at once, but interleaved.
- AnotherGoodName 3y agoAgreed. I think people would accept the <50% perf as long as there's plenty of RAM to play in. I think that's the biggest risk to Nvidia right now. No one's catching up to the topline perf or even power efficiency (Intel actually suck at perf/power right now) but they might be able to compete with raw GPU costs at least. Give us a $2k card with 256gb ram and we'll probably take it over the $40k 80gb cards despite the power efficiency and topline perf that the $40k card has.
- shsixhehgf 3y agoYou’re looking at it as if there’s some sort of technical bottleneck. Is not, it’s business. VRAM capacity is how they segregate data center/AI workloads versus gaming. It will be this way until they get a competitor.
- eldenring 3y agoYes it is quite expensive. The issue is memory bandwidth is a lot more constrained when youre routing to hundreds or thousands of cores instead of the handful you need to support on a CPU.
- ayakang31415 3y agoSo the higher cost from higher VRAM comes from the implementation of higher bandwidth to accommodate more data flow because there are more cores versus CPU?