2 ms·
> They are the first ones to ship on-device inference at scale on non-nvidia hardware Which is neat, but it's not CUDA. It's an application-specific accelerato
by talldayo 2y ago
> They are the first ones to ship on-device inference at scale on non-nvidia hardware
Which is neat, but it's not CUDA. It's an application-specific accelerator good at a small subset of operations, controlled by a high-level library the industry is unfamiliar with and too underpowered to run LLMs or image generators. The NPU is a novelty, and today's presentation more-or-less confirmed how useless it is for rich local-only operations.
> Apple also has the means to build data center training hardware using apple silicon if they want to do so.
They could, but that's not a competitor against an NVL72 with hundreds of terabytes of unified GPU memory. And then they would need a CUDA competitor, which could either mean reviving OpenCL's rotting corpse, adopting Tensorflow/Pytorch like a sane and well-reasoned company, or reinventing the wheel with an extra library/Accelerate Framework/MPS solution that nobody knows about and has to convert models to use.
So they can make servers, but Xserve showed us pretty clearly that you can lead a sysadmin to MacOS but you can't make them use it.
> they could also start to supply them with cloud inference hardware and strongarm them into only using apple servers to serve iOS requests.
I wonder how much money they would lose doing that, over just using the industry-standard Nvidia servers. Once you factor in the margins they would have made selling those chips as consumer systems, it's probably in the tens-of-millions.
- KeplerBoy 2y agoYou're approaching this from a developers point of view. Users absolutely don't care if their prompt response has been generated by a CUDA kernel or some poorly documented apple specific silicon a poor team at cupertino almost lost their sanity to while porting the model. And haven't they already spent quite a bit on money on their pytorch-like MLX framework?
- talldayo 2y ago> Users absolutely don't care if their prompt response has been generated by a CUDA kernel or some poorly documented apple specific silicon They most certainly will. If you run GPT-4o on an iPhone with MLX, it will suck. Users will tell you it sucks, and they won't do so in developer-specific terms. The entire point of this thread is that Apple can't make users happy with their Neural Engine. They require a stopgap cloud solution to make up for the lack of local power on iPhone. > And haven't they already spent quite a bit on money on their pytorch-like MLX framework? As well as Accelerate Framework, Metal Performance Shaders and previously, OpenCL. Apple can't decide where to focus their efforts, least of which in a way that threatens CUDA as a platform.
- PartiallyTyped 2y agoImho, the stronghold of cuda is slowly eroding. Inference can run without it, and could so for years via ONNX. Now we are starting to see more back-ends becoming available. see https://github.com/openxla https://github.com/openxla
- alextheparrot 2y agoBit of a detail, but where are you deriving “with hundreds of terabytes of unified GPU memory” from?
- talldayo 2y agoI was an order of magnitude off, at least in the case of NVL72: https://www.nvidia.com/en-us/data-center/gb200-nvl72/ https://www.nvidia.com/en-us/data-center/gb200-nvl72/ But the point stands, these systems occupy a niche that Apple Silicon is poorly suited to filling. They run normal Linux, they support common APIs, and network to dozens of other machines using Infiniband.
- Miraste 2y ago> reinventing the wheel with an extra library/Accelerate Framework/MPS solution that nobody knows about and has to convert models to use. This is Apple's favorite thing in the world. They already have an Apple-Silicon-only ML framework as of a few months ago, called MLX. Does anyone know about it? No. Do you need to convert models to use it? Yes.