7 ms·
Since no one specifically answered your question yet, yes, you should be able to get usable performance. A Q4_K_M GGUF of DeepSeek-R1 is 404GB. This is a 671B M
by lhl 2y ago
Since no one specifically answered your question yet, yes, you should be able to get usable performance. A Q4_K_M GGUF of DeepSeek-R1 is 404GB. This is a 671B MoE that "only" has 37B activations per pass. You'd probably expect in the ballpark of 20-30 tok/s (depends on how much actually MBW can be utilized) for text generation.
From my napkin math, the M3 Ultra TFLOPs is still relatively low (around 43 FP16 TFLOPs?), but it should be more than enough to handle bs=1 token generation (should be way <10 FLOPs/byte for inference). Now as far is its prefill/prompt processing speed... well, that's another matter.
- drited 2y agoI would be curious about context window size that would be expected when generating ballpark 20 to 20 tokens per second using Deepseek-R1 Q4 on this hardware?
- lynguist 2y agoI actually think it’s not a coincidence and they specifically built this M3 Ultra for DeepSeek R1 4-bit. They also highlight in their press release that they tested it with 600B class LLMs (DeepSeek R1 without referring to it by name). And they specifically did not stop at 256 GB RAM to make this happen. Maybe I’m reading too much into it.
- forrestthewoods 2y agoI don’t think you understand hardware timelines if you think this product had literally anything to do with anything DeepSeek.
- bustling-noose 2y agoMy thoughts too. This product was in the pipeline maybe 2-3 years ago. Maybe with LLMs getting popular a year ago they tried to fit more memory but it’s almost impossible to do that that close to a launch. Especially when memory is fused not just a module you can swap.
- tgma 2y agoYour conclusion is correct but to be clear the memory is not "fused." It's soldered close to the main processor. Not even a Package-on-Package (two story) configuration. See photo without heatspreader here: https://wccftech.com/apple-m2-ultra-soc-delidded-package-size-bigger-than-intel-sapphire-rapids-xeon-cpu/ https://wccftech.com/apple-m2-ultra-soc-delidded-package-siz...
- bustling-noose 2y agoI think by fuse I mean't its stuck on to the SOC module, not part of the SOC as I may have worded. While you could maybe still add NANDs later in the manufacturing process, it's probably not easy, especially if you need more NANDs and a larger module which might cause more design problems. The NAND is closer cause the controller is in the SOC. So the memory controller probably would also change with higher memory sizes which would mean this cannot be a last minute change.
- fennecfoxy 2y agoSheesh, the...comments on that link.
- reitzensteinm 2y agoChip? Yes. Product? Not necessarily... It's not completely out of the question that the 512gb version of M3 Ultra was built for their internal Apple silicon servers powering Private Compute Cloud, but not intended for consumer release, until a compelling use case suddenly arrived. I don't _think_ this is what happened, but I wouldn't go as far as to call it impossible.
- forrestthewoods 2y agoDeepSeek R1 came out Jan 20. Literally impossible.
- reitzensteinm 2y agoThe scenario is that the 512gb M3 Ultra was validated for the Mac Studio, and in volume production for their servers, but a business decision was made to not offer more than a 256gb SKU for Mac Studio. I don't think this happened, but it's absolutely not "literally impossible". Engineering takes time, artificial segmentation can be changed much more quickly.
- forrestthewoods 2y agoFrom “internal only” to “delivered to customers” in 6 weeks is literally impossible.
- ryao 2y agoThis change is mostly just using higher density ICs on the assembly line and printing different box art with a SKU change. It does not take much time, especially if they had planned it as a possible product just in case management changed its mind.
- jahewson 2y agoThat's absurd. Fabing custom silicon is not something anybody does for a few thousand internal servers. The unit economics simply don't work. Plus Apple is using OpenAI to provide its larger models anyway, so the need never even existed.
- a1o 2y agoAny ideas on power consumption? I wonder how much power would that use. It looks like it would be more efficient than everything else that currently exists.
- ryao 2y agoThe M2 Ultra Mac Pro could reach a maximum of 330W according to Apple: https://support.apple.com/en-us/102839 https://support.apple.com/en-us/102839 I assume it is similar.
- j45 2y agoLooks like up to 480W listed here https://www.apple.com/mac-studio/specs/ https://www.apple.com/mac-studio/specs/
- a1o 2y agoThanks!!
- kmacdough 2y agoThat or it's the luckiest coincidence! In all seriousness, Apple is fairly consistent about not pushing specs that don't matter and >256GB is just unnecessary for most other common workloads. Factors like memory bandwidth, core count and consumption/heat would have higher impact. That said, I doubt it was explicitly for R1, but rather based the industry a few years ago when GPT 3s 170B was SOTA, but the industry was still looking larger. "As much memory as possible" is the name of the game for AI in a way that's not true for other workloads. It may not be true for AI forever either.
- icedchai 2y agoThe high end Intel Macs supported over a TB of RAM, over 5 years ago. It's kinda crazy Apple's own high end chips didn't support more RAM. Also, the LLM use case isn't new... Though DeepSeek itself may be. RAM requirements always go up.
- teknologist 2y agoJust to clarify. There is an important difference between unified memory, meaning accessible by both CPU and GPU, and regular RAM that is only accessible by CPU.
- angoragoats 2y agoAs mentioned elsewhere in this thread, unified memory has existed long before Apple released the M1 CPU, and in fact many Intel processors that Apple used before supported it (though the Mac pros that supported 1.5TB of RAM did not, as they did not have integrated graphics). The presence of unified memory does not necessarily make a system better. It’s a trade off: the M-series systems have high memory bandwidth thanks to the large number of memory channels, and the integrated GPUs are faster than most others. But you can’t swap in a faster GPU, and when using large LLMs even a Mac Studio is quite slow compared to using discrete GPUs.
- nightski 2y ago$10k to run a 4 bit quantized model. Ouch.
- vaxman 2y ago[flagged]
- vaxman 2y ago[flagged]
- 1R053 2y agoare you comparing the same models? How did you calculate the TOPS for M3 Ultra?
- vaxman 2y agoAn M3 Ultra is two M3 Max chips connected via fabric, so physics. Did not mean to shit on anyone's parade, but it's a trap for novices, with the caveat that you reportedly can't buy a GB10 until "May 2025" and the expectation that it will be severely supply constrained. For some (overfunded startups running on AI monkey code? Youtube Influencers?), that timing is an unacceptable risk, so I do expect these things to fly off the shelves and then hit eBay this Summer.
- fredoliveira 2y agowhat?
- vaxman 2y agoSorry, an apostrophe got lost in "PO's"
- titanomachy 2y agoI'm downvoting you because your use of language is so annoying, not because I work for Apple.
- tgma 2y agoPretty sure this has absolutely nothing to do with Deepseek and even local LLM at large, which has been a thing for a while and an obvious use case original Llama leak and llama.cpp coming around. Fact is Mac Pros in the Intel days supported 1.5TB RAM in some configurations[1] and that was 6 years ago expectations of their high end customer base. They needed to address the gap for those customers so they would have shipped such a product regardless. Local LLM is cherry-on-top. Deepseek in particular almost certainly had nothing to do with it. They will still need to double their supported RAM in their SoC to get there. Perhaps in a Mac Pro or a different quad-Max-glued chip. [1]: https://support.apple.com/en-us/101639 https://support.apple.com/en-us/101639
- khana 2y ago[dead]
- brookst 2y agoDesign work on the Ultra would have started 2-3 years ago, and specs for memory at least 18 months ago. I’m not sure they had that kind of inside knowledge for what Deepseek specifically was doing that far in advance. Did Deepseek even know that long ago?
- happyopossum 2y ago> they specifically built this M3 Ultra for DeepSeek R1 4-bit Which came out in what, mid January? Yeah, there's no chance Apple (or anyone) has built a new chip in the last 45 days.
- SV_BubbleTime 2y agoNo one is saying they built a new chip. But the decision to come to market with a 512GB sku may have changed from not making sense to “people will buy this”.
- deleted 2y ago[deleted]
- cyanydeez 2y agoDies are designed in years. This was just a coincidence.
- SV_BubbleTime 2y agoWhat part of “no one is saying they designed a new chip” is lost here?
- cyanydeez 2y agoSorry, non of us a fan boys trying to shape apple is great narratives
- vaxman 2y ago"No chance?" But it has been reported that the next generation of Apple Silicon started production a few weeks ago. Those deliveries may enable Apple to release its remaining M3 Ultra SKUs for sale to the public (because it has something Better for its internal PCC build-out). It also may point to other devices ᯅ depending upon such new Apple Silicon arriving sooner, rather than later. (Hey, I should start a YouTube channel or religion or something. /s)
- 2y ago
- jrflowers 2y ago> they specifically built this M3 Ultra for DeepSeek R1 4-bit. This makes sense. They started gluing M* chips together to make Mac Studios three years ago, which must have been in anticipation of DeepSeek R1 4-bit
- deleted 2y ago[deleted]