4 ms·

I run 4 Mac Studio ultras at work (they’re pricy when maxed out), for local-first AI dev services. But there’s a few things that make me want to switch to the S

I run 4 Mac Studio ultras at work (they’re pricy when maxed out), for local-first AI dev services. But there’s a few things that make me want to switch to the Spark. Networking is the biggest one, the Macs have Thunderbolt and Ethernet, but if I run distributed inference with EXO over Thunderbolt; the drop in tokens/second is massive. These Sparks get RDMA and can stack nicely. The other big one is access to CUDA, MLX has come a long way but being able to have CUDA and GPU access in containers would simplify the stack so nicely. If I had a USB-C/Thunderbolt backplane it might compare, but scaling with the Spark is likely a lot more straightforward. I call the stack with Mac Studios “MacAIver” because it feels like a duct tape solution, but the Spark equivalent would likely be more elegant.

aurareturn 1y ago
You'd have to stack 16 of these to get 2TB of VRAM, equivalent to 4 Mac Studios 512GBs chained together. 16 compared to 4. Surely even much faster networking in the Spark would degrade with that many devices? Biggest problem with Macs is that they don't have dedicated tensor cores in the GPU which makes prompt processing very slow compared to Nvidia and AMD.
- themgt 1y ago
  n.b. there's been a little speculation that Apple adding TensorOps to Metal 4 suggests M5/M6 may get tensor cores. https://x.com/liuliu/status/1932158994698932505 https://x.com/liuliu/status/1932158994698932505 https://developer.apple.com/metal/Metal-Shading-Language-Specification.pdf https://developer.apple.com/metal/Metal-Shading-Language-Spe...
  - aurareturn 1y ago
    Nice. I hope so. That would make Macs the best local LLM machines for the masses by far.
- thomasskis 1y ago
  It’s $12k for each Mac Studio, and the networking makes them only effective individually (it’s like less that 15 tokens/s with EXO) while NVLINK is very effective. The Spark is definitely more scalable, but the MLX and metal teams are cooking, so honestly either way is still winning.

aurareturn 1y ago

You'd have to stack 16 of these to get 2TB of VRAM, equivalent to 4 Mac Studios 512GBs chained together. 16 compared to 4. Surely even much faster networking in the Spark would degrade with that many devices? Biggest problem with Macs is that they don't have dedicated tensor cores in the GPU which makes prompt processing very slow compared to Nvidia and AMD.

themgt 1y ago
n.b. there's been a little speculation that Apple adding TensorOps to Metal 4 suggests M5/M6 may get tensor cores. https://x.com/liuliu/status/1932158994698932505 https://x.com/liuliu/status/1932158994698932505 https://developer.apple.com/metal/Metal-Shading-Language-Specification.pdf https://developer.apple.com/metal/Metal-Shading-Language-Spe...
- aurareturn 1y ago
  Nice. I hope so. That would make Macs the best local LLM machines for the masses by far.
thomasskis 1y ago
It’s $12k for each Mac Studio, and the networking makes them only effective individually (it’s like less that 15 tokens/s with EXO) while NVLINK is very effective. The Spark is definitely more scalable, but the MLX and metal teams are cooking, so honestly either way is still winning.