5 ms·
At this point the M3 Max in a MacBook Pro with 96/128GB V/RAM needs to be benched against various Nvidia cards like 4090 & the H100 for $/performance numbers
by iAkashPaul 3y ago
At this point the M3 Max in a MacBook Pro with 96/128GB V/RAM needs to be benched against various Nvidia cards like 4090 & the H100 for $/performance numbers
- NavinF 3y agoWhy the H100? Comparing against similarly priced consumer gear (3x 4090) makes a lot more sense, but I think you already know that one of these systems has an order of magnitude better perf/$ than the other
- iAkashPaul 3y agoYeah even that as well
- NavinF 3y agoNot sure if you missed my point. 3x 4090 will be 10x faster while costing the same. There's no point making the comparison. (Assuming both machines are running the same model. I see a lotta crazy numbers thrown around based on comparing very different quantizations)
- iAkashPaul 3y agoINT4 or FP4 ones are of interest to me
- ianpurton 3y agoRunning LLama 2 70b a Mac Studio gets 13 tokens per second. 2 x 4090 is more like 38 tokens per second. But it's harder than that. As the 2 x 4090 is using https://llm.mlc.ai/ https://llm.mlc.ai/ which is highly optimized so perhaps the Mac Studio inference could be better when tuned. Then a lot of production inference engines use continous batching, which means they perform really well when processing multiple prompts at a time.
- brucethemoose2 3y agoYou can squeeze 70B onto a single 4090/3090 with exllamav2. As of the latest 8 bit cache commit, I can manage 2.6bpw with seemingly good quality.
- taneq 3y agoWhat quantization are you talking about for these?
- brucethemoose2 3y agoThe sad thing is all of this hardware is outrageously overpriced. Its even sadder that AMD/Intel have failed to take advantage of the price gouging. They could have released affordable 32GB-48GB cards with (afaik) no silicon changes. The 7900 XTX is still somewhat reasonable where you can get it to work.
- iAkashPaul 3y agoTrue, Intel is trying its best at both hardware & recently software contributions like Neural Compressor, so let's see how far they take nvidia to task
- brucethemoose2 3y agoWell, they never released a 32GB A770. This is particularly perplexing, as they have no professional market to lose like AMD does. And the iGPUs are still pretty small. We will see how Meteor Lake and Battlemage shake out, but I don't like the delay rumors.
- jeroenhd 3y agoIf I remember correctly, Intel isn't making a lot of money on these GPUs. Their main goal for now is gaming, and you don't need 32GB cards for that. If they intend to release an AI card, I bet they'll charge through the nose for it, just like Nvidia does.
- brucethemoose2 3y agoWell, thats also unfortunate, as its in Intel's/AMD's interest to push into the local AI market with low pricing first.
- hypercube33 3y agoI just thought of this and the mobile chips with DDR would even work and probably be super affordable (not as fast) or even GDDR. Not sure how intel packages their iGPU and if it can do this as well - I have only followed the AMD side. I suppose nVidia could snag a lower end card and modify the memory interface and do the same thing quite easily using an older fab node.
- choppaface 3y agoFor inference or training? For training, Apple has no CUDA, so you're pretty much dead in the water, unless you're specifically interested in e.g. local fine-tuning (for e.g. privacy or start-up strategy of "use the user's compute"). For inference, 4090 and H100 are not designed for throughput per $. Nvidia GPUs like the A40 or T40 are tuned for cloud inference, but at the end of the day your product is either running inference server-side (so play your own CAPEX vs OPEX, maybe even try Jetson) or user-side and so the user chooses the hardware not you. I mean maybe there are Wallstreet analysts or people at Qualcomm/Broadcom who want an Apple vs Nvidia shootout to see who is getting the most of their silicon, but for the product space the comparison is basically irrelevant. Which is likely what Apple and Nvidia want right now.
- whywhywhywhy 3y agoYou might be disappointed when it comes to Stable Diffusion on M series it’s very slow compared to even 3090. Impressive it works at all but the difference is night and day compared to top of the line (gamer card) Nvidia. Like SDXL <7 seconds a generation vs 60+ seconds.
- Zetobal 3y agoIt's slow, badly support and doesn't scale like gaming. It's useless.