4 ms·

I wonder what Apple's priorities are for the M5 chip. I'm all in on inference and training in my personal life right now, so if I were deploying $20bn with TSMC

I wonder what Apple's priorities are for the M5 chip. I'm all in on inference and training in my personal life right now, so if I were deploying $20bn with TSMC, (or maybe more this year!), I guess I'd be trying to spec out a scalable architecture with as much high bandwidth memory as possible. I wonder if Apple's willing to spec out 350W datacenter chips or not, and if they're going to make a play at datacenter AI with this generation. It feels like it might be a good time to do it. Anything competitive at all will suck a little oxygen out of NVIDIA and probably pay for itself in market cap. Either way, I'm pleased my Apple tax is going to buy me 2nm chips next year or so; silicon feels very interesting in the next few years. Juicy.

talldayo 2y ago
> Anything competitive at all will suck a little oxygen out of NVIDIA and probably pay for itself in market cap. To whom? If you're Apple I guess it makes sense to invest in your own architecture. But Apple doesn't ship a CUDA equivalent, they don't support server OSes, and they make no attempt to support the few cross-platform solutions shipping today. In a lot of ways, they're even further behind than AMD's server-side AI offerings.
- vessenes 2y ago
  Apple ships MLX, which has found support in creator communities, and is starting to see support at places like hugging face. In fact, it seems to me to be growing faster than ROCM support, but of course, it is starting from scratch, and this is just anecdotal. And, MPS are a drop in replacement for many programs in pytorch -- just replacing device="cuda" with device="mps" works relatively well, much of the time. For those doing inference at the edge / playing / testing, mps works well -- it is entirely the reason for llama.cpp's popularity, for instance. Rumors are that Apple is looking at using high memory M2 chips for datacenter inference -- 192GB of RAM is pretty great, yo. So I guess I'm saying I think you're wrong on the facts -- Apple does support pytorch, it does have an alternate full stack offering with MLX, and I would not be surprised if more tokens are generated through llama.cpp's downstream offerings on Apple Silicon than are generated on AMD.
  - talldayo 2y ago
    MLX is not a cross-platform solution though, and it's certainly not a one-size-fits-all GPGPU SDK a-la CUDA. It's a nice acceleration option on top of Accelerate framework and MPS but it's not going to kill Nvidia (or probably even steal market-share; it's local versus server we're talking about here). At best, it's another executor to throw onto the ONNX pile of Competing Standards: https://onnxruntime.ai/docs/execution-providers/ https://onnxruntime.ai/docs/execution-providers/ > Rumors are that Apple is looking at using high memory M2 chips for datacenter inference -- 192GB of RAM is pretty great, yo. The only thing better than 192GB of coherent memory is 384 GB HBM3e networked to other machines at 16tb/s: https://www.nvidia.com/en-us/data-center/gb200-nvl72/ https://www.nvidia.com/en-us/data-center/gb200-nvl72/ It also helps that Nvidia has deep industry buy-in and supports the server market much more than Apple does. But strictly comparing Apple's potential server offerings to Nvidia's Grace systems is a blowout; Apple genuinely needs custom hardware to stand out in the server market. It's my opinion that Apple just can't do this. They're an entire TSMC node ahead of Nvidia and still are getting smoked in performance-per-watt by their consumer-grade chips: https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks
    - spacedcowboy 2y ago
      A gb200-nvl72 seems to cost [1] about $3,000,000. I expect Apple could throw together a few, several even, 192GB M2's for that sort of money - and they may not have a 16TB bus, but I suspect (for the same price) their quantity might make up for it in the problem-sphere that a gb200-nvl72 would expect to be in... [1] https://www.tweaktown.com/news/98292/nvidias-new-gb200-superchip-costs-up-to-70-000-full-b200-nvl72-ai-server-3-million/index.html https://www.tweaktown.com/news/98292/nvidias-new-gb200-super...
      - talldayo 2y ago
        It's expensive, but that's because 72 GB200s are faster than 144 networked M2 Ultras. You're tackling bigger workloads with better-standardized software, genuine first-party support and proprietary integration that goes even further than Apple's memory coherency. Datacenter customers aren't drooling over the advantages of Apple Silicon because most of them are table-stakes at their scale; Apple has to go the extra mile to stand out, specifically for these customers. I won't deny, Nvidia charges the sort of margins that would make Apple blush. But they do that (and are successful) because they're scaling the workloads Apple won't. Apple turned their back on Nvidia, gave up funding Kronos' alternative to CUDA, and now wants a piece of the pie they've spent the past 15 years trying to avoid. It just doesn't seem like they're capable of winning here; their strategy is to dominate or die, and domination isn't an option.

talldayo 2y ago

> Anything competitive at all will suck a little oxygen out of NVIDIA and probably pay for itself in market cap. To whom? If you're Apple I guess it makes sense to invest in your own architecture. But Apple doesn't ship a CUDA equivalent, they don't support server OSes, and they make no attempt to support the few cross-platform solutions shipping today. In a lot of ways, they're even further behind than AMD's server-side AI offerings.

vessenes 2y ago
Apple ships MLX, which has found support in creator communities, and is starting to see support at places like hugging face. In fact, it seems to me to be growing faster than ROCM support, but of course, it is starting from scratch, and this is just anecdotal. And, MPS are a drop in replacement for many programs in pytorch -- just replacing device="cuda" with device="mps" works relatively well, much of the time. For those doing inference at the edge / playing / testing, mps works well -- it is entirely the reason for llama.cpp's popularity, for instance. Rumors are that Apple is looking at using high memory M2 chips for datacenter inference -- 192GB of RAM is pretty great, yo. So I guess I'm saying I think you're wrong on the facts -- Apple does support pytorch, it does have an alternate full stack offering with MLX, and I would not be surprised if more tokens are generated through llama.cpp's downstream offerings on Apple Silicon than are generated on AMD.
- talldayo 2y ago
  MLX is not a cross-platform solution though, and it's certainly not a one-size-fits-all GPGPU SDK a-la CUDA. It's a nice acceleration option on top of Accelerate framework and MPS but it's not going to kill Nvidia (or probably even steal market-share; it's local versus server we're talking about here). At best, it's another executor to throw onto the ONNX pile of Competing Standards: https://onnxruntime.ai/docs/execution-providers/ https://onnxruntime.ai/docs/execution-providers/ > Rumors are that Apple is looking at using high memory M2 chips for datacenter inference -- 192GB of RAM is pretty great, yo. The only thing better than 192GB of coherent memory is 384 GB HBM3e networked to other machines at 16tb/s: https://www.nvidia.com/en-us/data-center/gb200-nvl72/ https://www.nvidia.com/en-us/data-center/gb200-nvl72/ It also helps that Nvidia has deep industry buy-in and supports the server market much more than Apple does. But strictly comparing Apple's potential server offerings to Nvidia's Grace systems is a blowout; Apple genuinely needs custom hardware to stand out in the server market. It's my opinion that Apple just can't do this. They're an entire TSMC node ahead of Nvidia and still are getting smoked in performance-per-watt by their consumer-grade chips: https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks
  - spacedcowboy 2y ago
    A gb200-nvl72 seems to cost [1] about $3,000,000. I expect Apple could throw together a few, several even, 192GB M2's for that sort of money - and they may not have a 16TB bus, but I suspect (for the same price) their quantity might make up for it in the problem-sphere that a gb200-nvl72 would expect to be in... [1] https://www.tweaktown.com/news/98292/nvidias-new-gb200-superchip-costs-up-to-70-000-full-b200-nvl72-ai-server-3-million/index.html https://www.tweaktown.com/news/98292/nvidias-new-gb200-super...
    - talldayo 2y ago
      It's expensive, but that's because 72 GB200s are faster than 144 networked M2 Ultras. You're tackling bigger workloads with better-standardized software, genuine first-party support and proprietary integration that goes even further than Apple's memory coherency. Datacenter customers aren't drooling over the advantages of Apple Silicon because most of them are table-stakes at their scale; Apple has to go the extra mile to stand out, specifically for these customers. I won't deny, Nvidia charges the sort of margins that would make Apple blush. But they do that (and are successful) because they're scaling the workloads Apple won't. Apple turned their back on Nvidia, gave up funding Kronos' alternative to CUDA, and now wants a piece of the pie they've spent the past 15 years trying to avoid. It just doesn't seem like they're capable of winning here; their strategy is to dominate or die, and domination isn't an option.