3 ms·
You’re largely correct for training models Where this hardware shines is inference (aka developing products on top of the models themselves)
by DrBenCarson 2y ago
You’re largely correct for training models
Where this hardware shines is inference (aka developing products on top of the models themselves)
- 827a 2y agoTrue. But with Project Digits supposedly around the corner, which supposedly costs $3,000 and supports ConnectX and runs Blackwell; what's the over-under on just buying two of those at about half the price of one maxed M3 Ultra Mac Studio?
- DrBenCarson 2y agoAnd how much VRAM will Project Digits have?
- 827a 2y ago128gb each, so two would have 256gb. Its half that of a max spec Mac Studio, but also half the price and eight times faster memory speed. Realistically which open source LLMs does 512gb over 256gb of memory unlock? My understanding is that the true bleeding edge ones like R1 won't even handle 512gb well, especially with the anemic memory speed.
- seanmcdirmid 2y agoWe really should see what happens when Project Digits is finally released. Also, I would love in NVIDIA decided to get in the CPU/GPU + unified memory space. I can't imagine the M3 Ultra doing well on a model that loads into ~500G, but they should be a blast on 70b models (well, twice as fast as my M3 Max at least) or even a heavily quantized 400b model.
- DrBenCarson 2y agoI agree project digits looks to be the better all-around option for AI researchers, but I still think the Mac is better for people building products with AI Re memory speed, digits will be at 273GB/s while the Mac Studio is at 819GB/s Not to mention the Mac has 6 120GB/s thunderbolt 5 ports and can easily be used for video editing, app development, etc.