5 ms·

Not sure Nvidia is too happy with Apple. They are the first ones to ship on-device inference at scale on non-nvidia hardware. Apple also has the means to build

Not sure Nvidia is too happy with Apple. They are the first ones to ship on-device inference at scale on non-nvidia hardware. Apple also has the means to build data center training hardware using apple silicon if they want to do so. If they are serious about the OAI partnership they could also start to supply them with cloud inference hardware and strongarm them into only using apple servers to serve iOS requests.

whimsicalism 2y ago
> Apple also has the means to build data center training hardware using apple silicon if they want to do so. i'm seeing people all over this thread saying stuff like that, it reads like fantasyland to me. Apple doesn't have the talent or the chips or suppliers or really any of the capabilities to do this, where are people getting it from?
- KeplerBoy 2y ago
  Apple is already one of the largest (if not the largest) customers of TSMC and they have plenty of experience designing some of the best chips on the most modern nodes. Their ability to design a chip and networking fabric which is fast/efficient at training a narrow set of model architecture is not far fetched by any means.
  - talldayo 2y ago
    It's worth noting that one of Apple's largest competitor at TSMC is, in fact, Nvidia. And when you line the benchmarks up, Nvidia is one of the few companies that consistently beats Apple on performance-per-watt even when they aren't on the same TSMC node: https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks
mistersquid 2y ago
> Apple also has the means to build data center training hardware using apple silicon if they want to do so. > If they are serious about the OAI partnership they could also start to supply them with cloud inference hardware and strongarm them into only using apple servers to serve iOS requests Apple addressed both these points in today’s preso. 1. They will send requests that require larger contexts to their own Apple Silicon-based servers that will provide Apple devices a new product platform called Private Cloud Compute. 2. Apple’s OS generative AI request APIs won’t even talk to cloud compute resources that do not attest to infrastructure that has a publicly available privacy audit.
- wmf 2y ago
  I'm pretty sure those points do not apply to ChatGPT integration. ChatGPT is still running on Nvidia.
  - mistersquid 2y ago
    > I'm pretty sure those points do not apply to ChatGPT integration. You’re absolutely right. I got too excited about Apple’s strategy to encourage developers to use Apple Private Cloud Compute. The UX for ChatGPT as shown for iOS 18 makes it obvious that you are sending data outside the Apple Silicon walled garden.
talldayo 2y ago
> They are the first ones to ship on-device inference at scale on non-nvidia hardware Which is neat, but it's not CUDA. It's an application-specific accelerator good at a small subset of operations, controlled by a high-level library the industry is unfamiliar with and too underpowered to run LLMs or image generators. The NPU is a novelty, and today's presentation more-or-less confirmed how useless it is for rich local-only operations. > Apple also has the means to build data center training hardware using apple silicon if they want to do so. They could, but that's not a competitor against an NVL72 with hundreds of terabytes of unified GPU memory. And then they would need a CUDA competitor, which could either mean reviving OpenCL's rotting corpse, adopting Tensorflow/Pytorch like a sane and well-reasoned company, or reinventing the wheel with an extra library/Accelerate Framework/MPS solution that nobody knows about and has to convert models to use. So they can make servers, but Xserve showed us pretty clearly that you can lead a sysadmin to MacOS but you can't make them use it. > they could also start to supply them with cloud inference hardware and strongarm them into only using apple servers to serve iOS requests. I wonder how much money they would lose doing that, over just using the industry-standard Nvidia servers. Once you factor in the margins they would have made selling those chips as consumer systems, it's probably in the tens-of-millions.
- KeplerBoy 2y ago
  You're approaching this from a developers point of view. Users absolutely don't care if their prompt response has been generated by a CUDA kernel or some poorly documented apple specific silicon a poor team at cupertino almost lost their sanity to while porting the model. And haven't they already spent quite a bit on money on their pytorch-like MLX framework?
  - talldayo 2y ago
    > Users absolutely don't care if their prompt response has been generated by a CUDA kernel or some poorly documented apple specific silicon They most certainly will. If you run GPT-4o on an iPhone with MLX, it will suck. Users will tell you it sucks, and they won't do so in developer-specific terms. The entire point of this thread is that Apple can't make users happy with their Neural Engine. They require a stopgap cloud solution to make up for the lack of local power on iPhone. > And haven't they already spent quite a bit on money on their pytorch-like MLX framework? As well as Accelerate Framework, Metal Performance Shaders and previously, OpenCL. Apple can't decide where to focus their efforts, least of which in a way that threatens CUDA as a platform.
wmf 2y ago
I would say MS Copilot+ is shipping on-device inference a few months before Apple, although at 1000x lower volume.

whimsicalism 2y ago

> Apple also has the means to build data center training hardware using apple silicon if they want to do so. i'm seeing people all over this thread saying stuff like that, it reads like fantasyland to me. Apple doesn't have the talent or the chips or suppliers or really any of the capabilities to do this, where are people getting it from?

KeplerBoy 2y ago
Apple is already one of the largest (if not the largest) customers of TSMC and they have plenty of experience designing some of the best chips on the most modern nodes. Their ability to design a chip and networking fabric which is fast/efficient at training a narrow set of model architecture is not far fetched by any means.
- talldayo 2y ago
  It's worth noting that one of Apple's largest competitor at TSMC is, in fact, Nvidia. And when you line the benchmarks up, Nvidia is one of the few companies that consistently beats Apple on performance-per-watt even when they aren't on the same TSMC node: https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks

mistersquid 2y ago

> Apple also has the means to build data center training hardware using apple silicon if they want to do so. > If they are serious about the OAI partnership they could also start to supply them with cloud inference hardware and strongarm them into only using apple servers to serve iOS requests Apple addressed both these points in today’s preso. 1. They will send requests that require larger contexts to their own Apple Silicon-based servers that will provide Apple devices a new product platform called Private Cloud Compute. 2. Apple’s OS generative AI request APIs won’t even talk to cloud compute resources that do not attest to infrastructure that has a publicly available privacy audit.

wmf 2y ago
I'm pretty sure those points do not apply to ChatGPT integration. ChatGPT is still running on Nvidia.
- mistersquid 2y ago
  > I'm pretty sure those points do not apply to ChatGPT integration. You’re absolutely right. I got too excited about Apple’s strategy to encourage developers to use Apple Private Cloud Compute. The UX for ChatGPT as shown for iOS 18 makes it obvious that you are sending data outside the Apple Silicon walled garden.

talldayo 2y ago

> They are the first ones to ship on-device inference at scale on non-nvidia hardware Which is neat, but it's not CUDA. It's an application-specific accelerator good at a small subset of operations, controlled by a high-level library the industry is unfamiliar with and too underpowered to run LLMs or image generators. The NPU is a novelty, and today's presentation more-or-less confirmed how useless it is for rich local-only operations. > Apple also has the means to build data center training hardware using apple silicon if they want to do so. They could, but that's not a competitor against an NVL72 with hundreds of terabytes of unified GPU memory. And then they would need a CUDA competitor, which could either mean reviving OpenCL's rotting corpse, adopting Tensorflow/Pytorch like a sane and well-reasoned company, or reinventing the wheel with an extra library/Accelerate Framework/MPS solution that nobody knows about and has to convert models to use. So they can make servers, but Xserve showed us pretty clearly that you can lead a sysadmin to MacOS but you can't make them use it. > they could also start to supply them with cloud inference hardware and strongarm them into only using apple servers to serve iOS requests. I wonder how much money they would lose doing that, over just using the industry-standard Nvidia servers. Once you factor in the margins they would have made selling those chips as consumer systems, it's probably in the tens-of-millions.

KeplerBoy 2y ago
You're approaching this from a developers point of view. Users absolutely don't care if their prompt response has been generated by a CUDA kernel or some poorly documented apple specific silicon a poor team at cupertino almost lost their sanity to while porting the model. And haven't they already spent quite a bit on money on their pytorch-like MLX framework?
- talldayo 2y ago
  > Users absolutely don't care if their prompt response has been generated by a CUDA kernel or some poorly documented apple specific silicon They most certainly will. If you run GPT-4o on an iPhone with MLX, it will suck. Users will tell you it sucks, and they won't do so in developer-specific terms. The entire point of this thread is that Apple can't make users happy with their Neural Engine. They require a stopgap cloud solution to make up for the lack of local power on iPhone. > And haven't they already spent quite a bit on money on their pytorch-like MLX framework? As well as Accelerate Framework, Metal Performance Shaders and previously, OpenCL. Apple can't decide where to focus their efforts, least of which in a way that threatens CUDA as a platform.

wmf 2y ago

I would say MS Copilot+ is shipping on-device inference a few months before Apple, although at 1000x lower volume.