6 ms·
Nvidia's RTX 5000 Ada Now Available: AD102 with 32GB of GDDR6
- netdur 3y ago12,800 CUDA - $4,000.00
- valine 3y agoIt’s not a direct apples to apples comparison, but compared to a RTX 4090 this isn’t compelling. Other than the 32GB of VRAM there’s not much going for this card. For $4k most people would be better off with dual 4090s.
- 22c 3y agoPresumably the 32GB of VRAM is what makes it compelling, as you could cram some fairly substantial AI models on there.
- iaw 3y ago33% increase in VRAM over a consumer card at 150% increase in price? If CUDA is 100% compliant on AMD we'll actually see competition in this space.
- darklycan51 3y agoYeah, because price is a deciding factor for corporations, lol
- photonbeam 3y agoIn the long-term it is
- frognumber 3y agoNot really. If a SWE salary is $200k, total cost is probably around $400k. A $4k GPU, if bought each year, increases SWE cost by 1-2%. Buying a nice GPU was the best purchase my employer made, even if I use it every 6 months. It's already paid for itself many times over.
- PeterStuer 3y agoYou should never compare an additional cost to the overall c ost in a production setting. You should compare it to the profit margin. If you are running a 10% margin, then a 2% cost increase is -20% profit, a quite substantial amount.
- frognumber 3y agoThe key question is output per dollar. If employees are more efficient, I need fewer of them. People cost a lot of money relative to capital expenses like nice laptops, good chairs, or individual GPUs. This kind of spending almost always pays off very quickly.
- PeterStuer 3y agoYes, I also believe that. The only thing I argue against is the 'negligeable' marginal cost statement fed to devs by dev gurus because it is a fallacy. I'm all for investing in quality work experience. Decent equipment being the most obvious. Not only for the potential higher productivity, but also for its definite hr/retention effects. Investing in tangible production improvements and comfort is money spent 10x more effective than the hr semi-mandatory fun/social events that have metastasised throughout tech. Where asking wasn't an outcome, in contrast to some other colleague that just went "I guess we will just deliver below potential then" I used to buy my own stuff because I respected my time, but it definitely didn't want to keep me working there. Now this does not mean you have to shell out for every whim. If you want a 10.000€ PC setup instead of a 4.000€ one, I will need a better justification than 'custom loop and RGB'.
- 3y ago
- wkat4242 3y agoHave you worked at an enterprise? Everything is about money. Don't forget the scale at which these operate. $1000 extra in a GPU adds up quickly when you buy a thousand of them. Perhaps in big tech it's different but in our huge multinational anything IT is seen as a cost center only and has to be on the cheap. Which a real focus on short term savings because who knows if this year's SVP will still be in that seat next year. Nobody ever told me "let's take the most expensive option because it's the best in the long run" :(
- syntaxing 3y agoThat's kinda what top parent comment is talking about though. Money doesn't matter when you only have one choice. Enterprise can ONLY buy Nvidia cards so money really doesn't matter in this case. Technical execution is more important than anything else, including money. If you can't afford or make profit to buy cards to support what you need (more VRAM in this case), the project is dead in the water to begin with.
- valine 3y agoYou can split your AI models between two 4090s and have access to 48GB of VRAM for less money. You do take a slight performance hit compared to running on one card, but the 4090 also has more CUDA cores which help mitigate the slowdown.
- frognumber 3y agoIt has a few things going for it: 1) Enterprise-grade design, with features like ECC. I like ECC. 2) VRAM is basically the limiting factor on running LLMs. On my GPUs, I look at cost versus VRAM. Fitting a model in one PC helps. Fitting a model on one GPU helps more. 3) It's just over half the power (250W versus 450W). The enterprise cards are clocked a little bit slower, which dramatically reduces power. At least on the older Ampere cards, they also fit better. The consumer cards had giant cooling, which made sticking several in a computer more complex.
- aydyn 3y ago> The enterprise cards are clocked a little bit slower, which dramatically reduces power. But you can (and should) undervolt RTX 4090s if you care about power efficiency. Anyone buying one would know how to.
- stavros 3y agoDoes that get them to 250 W?
- syntaxing 3y agoNot OP, but you can limit the max wattage using nvidia-smi. Here's a good article with the instructions and how you would determine the best wattage. https://betterprogramming.pub/limiting-your-gpu-power-consumption-might-save-you-some-money-50084b305845 https://betterprogramming.pub/limiting-your-gpu-power-consum...
- stavros 3y agoThat's really interesting, thanks! I didn't realize you could do this though just an option.
- ksec 3y ago> I look at cost versus VRAM. Fitting a model in one PC helps. Fitting a model on one GPU helps more. Yes unfortunately I dont see 128GB VRAM coming any time soon even with 512bit GDDR Memory Interface.
- gmerc 3y agoThe 6000 makes a lot more sense than the 5000. Nvidia uses RAM limitations as non linear cost scaling vector in the same way openAI used context window but that opens opportunity for competition to disrupt long term.
- washadjeffmad 3y agoIt should bring pause to anyone considering buying an Apple Silicon device for inference, at least. And while you're right, I personally have no interest in running two 4090s. My 24GB 3090s and RTX 5500s can feel a little meager at times, but 4090s are still overpriced and power hungry. If I'm moving up, it'll be for density, and a few hundred won't matter.
- Dystopian 3y agoOr 2x 3090s to make it even more cost effective. If you're using these for purely inference (which is what the 5000 seems like it's tuned for) the vram's the real bottleneck so you get a similar bang for your buck using last gen's cards vs. the bleeding edge. I've also found the crypto crash has dumped a bunch of these well-worn cards on the market you can pick up a bit cheaper.
- hamilyon2 3y agoAbsolutely honest question, please consider me novice in the area. If you have $7K and want to run LLM locally, does getting Mac with M2 ultra and 192 Gb make most sense?
- kanwisher 3y agoFor inference it makes a lot of sense, but for training its going to be significantly slower then buying 2 3090s
- smoldesu 3y agoYou'll be able to load very large models, but it is far from the fastest inferencing option at $7,000. You can outperform the M2 Ultra's inferencing speed with an Nvidia machine half the price. It's not a bad pick necessarily, but there's also a good reason the M2 Ultra isn't the inferencing chip of choice in the industry; nobody really supports CoreML or Metal acceleration.
- MuffinFlavored 3y agoyou’ll be able to outperform on the same very large models or a $3.5k nvidia machine can’t load the very large models so it isn’t apples to apples comparison?
- smoldesu 3y agoWell, I mostly need an excuse to not bring the GH200 or Instict up and obsolete every consumer chip we've mentioned so far. Suffice to say that datacenters aren't exactly eying Apple Silicon for it's hardware value.
- syntaxing 3y agoI'm not entirely sure your statements are correct in today's landscape. I run models both on a M1 chip and 3090. The token generation are about the same if I use huggingface transformers on the 3090 or llama.cpp on the M1 with the same quantization (4 bit, I don't have enough RAM to run 8 bit on the M1). Most major local LLM libraries support Metal. I know you can use MPS backend on huggingface transformers but I have not tried it since llama.cpp (Ollama to be more specific) works great on a M1 to begin with.