3 ms·

Apple ships MLX, which has found support in creator communities, and is starting to see support at places like hugging face. In fact, it seems to me to be growi

Apple ships MLX, which has found support in creator communities, and is starting to see support at places like hugging face. In fact, it seems to me to be growing faster than ROCM support, but of course, it is starting from scratch, and this is just anecdotal. And, MPS are a drop in replacement for many programs in pytorch -- just replacing device="cuda" with device="mps" works relatively well, much of the time. For those doing inference at the edge / playing / testing, mps works well -- it is entirely the reason for llama.cpp's popularity, for instance. Rumors are that Apple is looking at using high memory M2 chips for datacenter inference -- 192GB of RAM is pretty great, yo. So I guess I'm saying I think you're wrong on the facts -- Apple does support pytorch, it does have an alternate full stack offering with MLX, and I would not be surprised if more tokens are generated through llama.cpp's downstream offerings on Apple Silicon than are generated on AMD.

talldayo 2y ago
MLX is not a cross-platform solution though, and it's certainly not a one-size-fits-all GPGPU SDK a-la CUDA. It's a nice acceleration option on top of Accelerate framework and MPS but it's not going to kill Nvidia (or probably even steal market-share; it's local versus server we're talking about here). At best, it's another executor to throw onto the ONNX pile of Competing Standards: https://onnxruntime.ai/docs/execution-providers/ https://onnxruntime.ai/docs/execution-providers/ > Rumors are that Apple is looking at using high memory M2 chips for datacenter inference -- 192GB of RAM is pretty great, yo. The only thing better than 192GB of coherent memory is 384 GB HBM3e networked to other machines at 16tb/s: https://www.nvidia.com/en-us/data-center/gb200-nvl72/ https://www.nvidia.com/en-us/data-center/gb200-nvl72/ It also helps that Nvidia has deep industry buy-in and supports the server market much more than Apple does. But strictly comparing Apple's potential server offerings to Nvidia's Grace systems is a blowout; Apple genuinely needs custom hardware to stand out in the server market. It's my opinion that Apple just can't do this. They're an entire TSMC node ahead of Nvidia and still are getting smoked in performance-per-watt by their consumer-grade chips: https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks
- spacedcowboy 2y ago
  A gb200-nvl72 seems to cost [1] about $3,000,000. I expect Apple could throw together a few, several even, 192GB M2's for that sort of money - and they may not have a 16TB bus, but I suspect (for the same price) their quantity might make up for it in the problem-sphere that a gb200-nvl72 would expect to be in... [1] https://www.tweaktown.com/news/98292/nvidias-new-gb200-superchip-costs-up-to-70-000-full-b200-nvl72-ai-server-3-million/index.html https://www.tweaktown.com/news/98292/nvidias-new-gb200-super...
  - talldayo 2y ago
    It's expensive, but that's because 72 GB200s are faster than 144 networked M2 Ultras. You're tackling bigger workloads with better-standardized software, genuine first-party support and proprietary integration that goes even further than Apple's memory coherency. Datacenter customers aren't drooling over the advantages of Apple Silicon because most of them are table-stakes at their scale; Apple has to go the extra mile to stand out, specifically for these customers. I won't deny, Nvidia charges the sort of margins that would make Apple blush. But they do that (and are successful) because they're scaling the workloads Apple won't. Apple turned their back on Nvidia, gave up funding Kronos' alternative to CUDA, and now wants a piece of the pie they've spent the past 15 years trying to avoid. It just doesn't seem like they're capable of winning here; their strategy is to dominate or die, and domination isn't an option.
    - spacedcowboy 2y ago
      An M2 system costs Apple a grand, two at most. You get a lot more than 144 of them for $3M. I don’t see how you get better support than from the company that built the hardware and software both… And I was under the impression that Apple was building these for themselves, they don’t need to satisfy other customers demands.
- vessenes 2y ago
  Obviously gb200 and m2 studios are very different price points and power envelopes, per other comments. I agree that Apple does not have a competitive server offering right now. But I think I'd still be spending toward that goal if I were reserving all the 2nm chips in the world next year. Your claim that NVIDIA has better performance per watt really surprised me, so I did some digging. My prior is that opencl is medium support at NVIDIA and very low support at Apple, so an OpenCL benchmark might not be fair. Reporting tokens/s, using llamafile's llama.cpp benchmarks for Mistral 7b f16 from Justine tunney's website (so this is non-batched): 192GB Mac Studio gives 79 prompt tok/sec and 15 eval tok/sec. https://www.baseten.co/blog/benchmarking-fast-mistral-7b-inference/ https://www.baseten.co/blog/benchmarking-fast-mistral-7b-inf... benchmarks an H100 on Mistral 7B, and claims different providers range from 38-170 TPS depending on their proprietary configurations. A used H100 server is roughly $80k according to google. H100 power draw alone is 700W, and with server is likely on the order of 1,000W. A new Mac Studio with 192GB of RAM is <$7k. The Studio power draw is max 295W. So, Apple in Cost / 150 tokens/s: $70,000. Power draw: 3kw. Nvidia in cost / 150 tokens/s using highly tuned proprietary software: $80k. Power draw: 1kw. Nvidia in cost / 150 tokens/s using Mistrals own cloud performance: $150k+, Power draw: 2kw-3kw. I don't think you're correct that Apples M2 chips get smoked by Nvidia's chips for these workloads.

talldayo 2y ago

MLX is not a cross-platform solution though, and it's certainly not a one-size-fits-all GPGPU SDK a-la CUDA. It's a nice acceleration option on top of Accelerate framework and MPS but it's not going to kill Nvidia (or probably even steal market-share; it's local versus server we're talking about here). At best, it's another executor to throw onto the ONNX pile of Competing Standards: https://onnxruntime.ai/docs/execution-providers/ https://onnxruntime.ai/docs/execution-providers/ > Rumors are that Apple is looking at using high memory M2 chips for datacenter inference -- 192GB of RAM is pretty great, yo. The only thing better than 192GB of coherent memory is 384 GB HBM3e networked to other machines at 16tb/s: https://www.nvidia.com/en-us/data-center/gb200-nvl72/ https://www.nvidia.com/en-us/data-center/gb200-nvl72/ It also helps that Nvidia has deep industry buy-in and supports the server market much more than Apple does. But strictly comparing Apple's potential server offerings to Nvidia's Grace systems is a blowout; Apple genuinely needs custom hardware to stand out in the server market. It's my opinion that Apple just can't do this. They're an entire TSMC node ahead of Nvidia and still are getting smoked in performance-per-watt by their consumer-grade chips: https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks

spacedcowboy 2y ago
A gb200-nvl72 seems to cost [1] about $3,000,000. I expect Apple could throw together a few, several even, 192GB M2's for that sort of money - and they may not have a 16TB bus, but I suspect (for the same price) their quantity might make up for it in the problem-sphere that a gb200-nvl72 would expect to be in... [1] https://www.tweaktown.com/news/98292/nvidias-new-gb200-superchip-costs-up-to-70-000-full-b200-nvl72-ai-server-3-million/index.html https://www.tweaktown.com/news/98292/nvidias-new-gb200-super...
- talldayo 2y ago
  It's expensive, but that's because 72 GB200s are faster than 144 networked M2 Ultras. You're tackling bigger workloads with better-standardized software, genuine first-party support and proprietary integration that goes even further than Apple's memory coherency. Datacenter customers aren't drooling over the advantages of Apple Silicon because most of them are table-stakes at their scale; Apple has to go the extra mile to stand out, specifically for these customers. I won't deny, Nvidia charges the sort of margins that would make Apple blush. But they do that (and are successful) because they're scaling the workloads Apple won't. Apple turned their back on Nvidia, gave up funding Kronos' alternative to CUDA, and now wants a piece of the pie they've spent the past 15 years trying to avoid. It just doesn't seem like they're capable of winning here; their strategy is to dominate or die, and domination isn't an option.
  - spacedcowboy 2y ago
    An M2 system costs Apple a grand, two at most. You get a lot more than 144 of them for $3M. I don’t see how you get better support than from the company that built the hardware and software both… And I was under the impression that Apple was building these for themselves, they don’t need to satisfy other customers demands.
vessenes 2y ago
Obviously gb200 and m2 studios are very different price points and power envelopes, per other comments. I agree that Apple does not have a competitive server offering right now. But I think I'd still be spending toward that goal if I were reserving all the 2nm chips in the world next year. Your claim that NVIDIA has better performance per watt really surprised me, so I did some digging. My prior is that opencl is medium support at NVIDIA and very low support at Apple, so an OpenCL benchmark might not be fair. Reporting tokens/s, using llamafile's llama.cpp benchmarks for Mistral 7b f16 from Justine tunney's website (so this is non-batched): 192GB Mac Studio gives 79 prompt tok/sec and 15 eval tok/sec. https://www.baseten.co/blog/benchmarking-fast-mistral-7b-inference/ https://www.baseten.co/blog/benchmarking-fast-mistral-7b-inf... benchmarks an H100 on Mistral 7B, and claims different providers range from 38-170 TPS depending on their proprietary configurations. A used H100 server is roughly $80k according to google. H100 power draw alone is 700W, and with server is likely on the order of 1,000W. A new Mac Studio with 192GB of RAM is <$7k. The Studio power draw is max 295W. So, Apple in Cost / 150 tokens/s: $70,000. Power draw: 3kw. Nvidia in cost / 150 tokens/s using highly tuned proprietary software: $80k. Power draw: 1kw. Nvidia in cost / 150 tokens/s using Mistrals own cloud performance: $150k+, Power draw: 2kw-3kw. I don't think you're correct that Apples M2 chips get smoked by Nvidia's chips for these workloads.