5 ms·
The salt is in the plain sight. The do the standard AMD comparison: 8x AMD MI300X (192GB, 750W) GPU 8x H100 SXM5 (80GB, 700W) GPU The fair comparison w
by nabla9 2y ago
The salt is in the plain sight.
The do the standard AMD comparison:
8x AMD MI300X (192GB, 750W) GPU
8x H100 SXM5 (80GB, 700W) GPU
The fair comparison would be against
8x H100 NVL (188GB, <800W) GPU
Price tells a story. If AMD performance would be in par with Nvidia they would not sell their cards for 1/4 price.
- fleischhauf 2y agoAMDs deep learning libraries are very bad the last time I checked, nobody uses amd in that space for that reason. Nvidia has a quazi monopoly, that's the main reason for the price difference IMHO.
- nabla9 2y ago> that's the main reason for the price difference IMHO. Explain why the performance difference does not matter? AMD does only 33% better with a chip that has 2X transistors and 2X memory.
- More-nitors 2y agothis... nearly 95% of deeplearning github repos are "tested using cuda gpu - others, not so sure" the only way to out-run nvidia is to have 3~10x better bang-for-buck. Or AMD can just provide a "DIY unlimited gpu RAM upgrade" kit -- a lot of people are buying macstudio 128gb ram because of its "bigger ram-for-buck" than nvidia gpus
- sroussey 2y agoI heard apple m4 ultra using 256gb HBM for studio and pro, but I don’t buy it. The 256GB maybe. But a HBM memory control that would go unused on laptops doesn’t pass the smell test.
- tracker1 2y agoI think their best option might be more/better prosumer options at the higher end of consumer pricing. Getting more hobbyists into play just on the value proposition.
- nabla9 2y agoMTr ------------------ H100 SXM5 80,000 MI300X 153,000 H100 NVL 160,000 H100 SXM4 has 52% of the transistors MI300X has, half of the RAM and MI300X achieves *ONLY* 33% higher throughput compared to the H100. MI300X was launched 6 months ago, H100 20 months ago. AMD has work to do.
- infamia 2y agoOn the other hand, 33% better performance for a 7% increase in power consumption is an appealing bullet point. Lots for AMD to do though, as you said.
- avidphantasm 2y agoMaybe I'm a naive fanboy, but I would put my money on Apple catching Nvidia before AMD or Intel.
- preisschild 2y agoBut Apple doesn't produce servers or server hardware.
- threecheese 2y agoGood point; if the Mx architecture does prove to be a viable competitor to Nvidia/AMD for training and/or inference, do you think Apple would enter the server market? They continue to diversify on the consumer side, I wonder if they have their eye on the business market; I am not sure if their general strategy of “prettier is better” would work well there though.
- sroussey 2y agoThey should buy Grok (the hardware inference company, not the lame twitter bot).
- bbatha 2y agoCurrently no, but the XServe was a product for a decade. And they have built an internal ML cloud, presumably with rack mountable hardware. The bigger issue for apple IMO is they ditched the server features of their OS and they're not going to sell a hypothetical M4ultra Xserve with linux.
- lhl 2y agoIsn't SXM5 higher bandwidth? It's 900 GB/s of bidirectional bandwidth per GPU across 18 NVLink 4 channels. The NVL's are on PCIe 5, and even w/ NVLink only get to 600 GB/s of bandwidth across 3 NVLink bridges (across only pairs of cards)? I haven't done a head to head and I suppose it depends on whether tensor parallelism actually scales linearly or not, but my understanding is since the NVL's are just PCIe/NVLink paired H100s, you're not really getting much if any benefit on something like vLLM. I think the more interesting thing critique might be the slightly odd choice of Mixtral 8x7B vs say a more standard Llama2/3 70B (or just test multiple models including some big ones like 8x22B or DBRX. Also, while I don't have a problem w/ vLLM, as TensorRT gets easier to set up, it might become a factor in comparisons (since they punted on FP8/AMP in this tests). Inferless published a shootoff a couple months ago comparing a few different inference engines: https://www.inferless.com/learn/exploring-llms-speed-benchmarks-independent-analysis https://www.inferless.com/learn/exploring-llms-speed-benchma... Price/perf does tell a story, but I think it's one that's mostly about Nvidia's platform dominance and profit margins more than intrinsic hardware advantages. On the spec sheet MI300X has a memory bandwidth and even raw FLOPS advantage but so far it has lacked proper software optimization/support and wide availability (has anyone besides hyperscalers and select partners been able to get them?)
- nabla9 2y ago> but I think it's one that's mostly about Nvidia's platform dominance and profit margins more Profit margins and dominance are result from performance, not the other way around. It does not matter if Nvidia tools are better when you deploy large number of chips for inference and it does more flops per watt or second. It's seller market and if AMD can't ask high price, their chip do not perform. ---- Question: People here seem to think that Nvidia has absolutely no advantage in their microarchitecture design skills. It's all in software or monopoly. Is this right?
- fisf 2y ago> People here seem to think that Nvidia has absolutely no advantage in their microarchitecture design skills. It's all in software or monopoly. That's an extrapolation. Microarchitecture design skills are not theoretical numbers you manage to put on a spec sheet. You cannot decouple the software driving the hardware - that's not a trivial problem.
- resource_waste 2y agoThx! Anyone who says Nivida isnt king, needs a reality check.
- ddtaylor 2y agoI love AMD, but my Nvidia stock position currently is much higher than AMD.
- szundi 2y agoThis topic is just about wether this changes or not
- hackerlight 2y agoAMD is at a much higher PE ratio. Is the market expecting AMD to up its game in the GPU sector? Or is the market expecting a pullback in GPU demand due to possibility for non-GPU AI solutions becoming the frontier or for AI investment to slow down?
- sangnoir 2y agoWhy not both?
- cameldrv 2y agoI think that the expectation is that NVIDIA is in somewhat of an unreasonable position right now (and for the immediate future) where they're getting about 80% gross margins on their datacenter GPUs. This is an extremely juicy target for competitors, and even if competitors manage to produce a product that's half as good as NVIDIA, NVIDIA will have to cut prices to compete.
- muxr 2y agoAMD doesn't have a higher PE ratio. You should use nonGAAP numbers because the GAAP numbers include Xilinx Googwill Amortization which skew the PE. AMD's PE is ~55. Nvidia's PE is above 70.
- sangnoir 2y ago> Price tells a story. If AMD performance would be in par with Nvidia they would not sell their cards for 1/4 price What were your thoughts on Zen (1) vs Intel's offerings then? AMD offered more back for the buck then too.
- ebalit 2y agoBut the price should be a factor. Your fair comparison would match a ~60k$ setup to a 20k$ according to prices we can find online. I don't think it should be ignored, especially when the power consumption is similar.
- fvv 2y agofair? h100 NVL are two h100 in a single package.. which probably costs 2xh100 or more, if so ok it's fair to compare 1 mi300x with 1 h100 NVL but then price ( and tco ) should be added to the some metrics conclusion , also the NVL is a 2xpci5.0 quad slot , so not the same thing.. I am not sure about system compatibility and if and how you can stack 8 of those in one system ( like you can do with non NVL and mi300x.. ) so it's a bit a diffent ( and more niche ) beast
- _dabs 2y agoPrice tells the story. Yes but for electric prices not card prize and here their much more close to each other!