4 ms·
> By the time you've shoehorned in an nVidia GPU and all that RAM, you're easily looking at 5x that MSRP That nvidia GPU setup will actually have the compute g
by kllrnohj 2y ago
> By the time you've shoehorned in an nVidia GPU and all that RAM, you're easily looking at 5x that MSRP
That nvidia GPU setup will actually have the compute grunt to make use of the RAM, though, which this M3 Ultra probably realistically doesn't. After all, if the only thing that mattered was RAM then the 2TB you can shove into an Epyc or Xeon would already be dominating the AI industry. But they aren't, because it isn't. It certainly hits at a unique combination of things, but whether or not that's maximally useful for the money is a completely different story.
- rbanffy 2y agoHad the M3 GPU been much wider, it would be constrained by the memory bandwidth. It might still have an advantage over Nvidia competitors in that it has 512GB accessible to it and will need to push less memory across socket boundaries. It all depends on the workload you want to run.
- stego-tech 2y agoYou're forgetting what Apple's been baking into their silicon for (nearly? over?) a decade: the Neural Processing Unit (NPU), now called the "Neural Engine". That's their secret sauce that makes their kit more competitive for endpoint and edge inference than standard x86 CPUs. It's why I can get similarly satisfying performance on my old M1 Pro Macbook Pro with a scant 16GB of memory as I can on my 10900k w/ 64GB RAM and an RTX 3090 under the hood. Just to put these two into context, I ran the latest version of LM Studio with the deepseek-r1-distill-llama-8b model @ Q8_0, both with the exact same prompt and maximally offloaded onto hardware acceleration and memory, with a context window that was entirely empty: Write me an AWS CloudFormation file that does the following: * Deploys an Amazon Kubernetes Cluster * Deploys Busybox in the namespace "Test1", including creating that Namespace * Deploys a second Busybox in the namespace "Test3", including creating that Namespace * Creates a PVC for 60GB of storage The M1Pro laptop with 16GB of Unified Memory: * 21.28 seconds for "Thinking" * 0.22s to the first token * 18.65 tokens/second over 1484 tokens in its responses * 1m:23s from sending the input to completion of the output The 10900k CPU, with 64GB of RAM and a full-fat RTX 3090 GPU in it: * 10.88 seconds for "thinking" * 0.04s to first token * 58.02 tokens/second over 1905 tokens in its responses * 0m:34s from sending the input to completion of the output Same model, same loader, different architectures and resources. This is why a lot of the AI crowd are on Macs: their chip designs, especially the Neural Engine and GPUs, allow quite competent edge inference while sipping comparative thimbles of energy. It's why if I were all-in on LLMs or leveraged them for work more often (which I intend to, given how I'm currently selling my generalist expertise to potential employers), I'd be seriously eyeballing these little Mac Studios for their local inference capabilities.
- kllrnohj 2y agoUh.... I must be missing something here, because you're hyping up Apple's NPU only to show it getting absolutely obliterated by the equally old 3090? Your 10900K having 64gb of RAM is also irrelevant here...
- stego-tech 2y agoYou're missing the the bigger picture by getting bogged down in technical details. To an end user, the difference between thirty seconds and ninety seconds is often irrelevant for things like AI, where they expect a delay while it "thinks". When taken in that context, you're now comparing a 14" laptop running off its battery, to a desktop rig gulping down ~500W according to my UPS, for a mere 66% reduction in runtime for a single query at the expense of 5x the power draw. Sure, the desktop machine performs better, as would a datacenter server jam-packed full of Blackwell GPUs, but that's not what's exciting about Apple's implementation. It's the efficiency of it all, being able to handle modern models on comparatively "weaker" hardware most folks would dismiss outright. That's the point I was trying to make.
- kllrnohj 2y agoWe're talking about the m3 ultra here, which is also wall powered and also expensive. Nobody is interested in dropping upwards of $10,000 on a Mac Studio to have "okay" performance just because an unrelated product is battery powered. Similarly saving a few bucks on electricity to triple the time the much, much more expensive engineer time spent waiting on results is foolish Also Apple isn't unique in having an NPU in a laptop. Fucking everyone does at this point.
- deleted 2y ago[deleted]