4 ms·

> Generative AI doesn't fit very well into Apple's hardware centric philosophy They have a huge unique niche to exploit with local private models, probably the

> Generative AI doesn't fit very well into Apple's hardware centric philosophy They have a huge unique niche to exploit with local private models, probably the only company on the planet with the hardware in place to take advantage of this today end-to-end, but unfortunately I don't think generative AI fits with Apple's culture for anything I just think they'd be terrified of it saying something wrong so would probably clip its wings to the point that it's mundane and kinda useless.

smoldesu 3y ago
I don't think it's a very huge or exploitable niche for two reasons: - Android, Windows, Linux and MacOS can already run local and private models just fine. Getting something product-ready for iPhone is a game of catch-up, and probably a losing battle if Apple insists on making you use their AI assistant over competing options. - The software side needs more development. The current SOTA inferencing techniques for Apple Silicon in llama.cpp are cobbled together with Metal shaders and NEON instructions, neither of which are ideal or specific to Apple hardware. If Apple wants to differentiate their silicon from the 2016 Macbooks running LLaMA with AVX, then they have to develop CoreML's API further.
- capableweb 3y ago
  Not saying you're right/wrong, but argument from the other side: Apple's hardware ships with hardware specifically for ML, and a lot of it (maybe even all of the recently released?) has shared memory between GPU/CPU, meaning you could run larger models. Windows/Linux is currently stuck with having to run inference with either RAM or VRAM, limiting the options quite a bit.
- jroesch 3y ago
  Things are already possible on today's hardware, see https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/mlc-llm which allows many models to be run on M1/M2 Macs, WASM, iOS and more. The main limiting factor will be small enough, high quality enough models that performance is high enough ultimately this is HW limited and they will need to improve the neural engine/map more computation on to it to make the mobile exp. possible.

smoldesu 3y ago

I don't think it's a very huge or exploitable niche for two reasons: - Android, Windows, Linux and MacOS can already run local and private models just fine. Getting something product-ready for iPhone is a game of catch-up, and probably a losing battle if Apple insists on making you use their AI assistant over competing options. - The software side needs more development. The current SOTA inferencing techniques for Apple Silicon in llama.cpp are cobbled together with Metal shaders and NEON instructions, neither of which are ideal or specific to Apple hardware. If Apple wants to differentiate their silicon from the 2016 Macbooks running LLaMA with AVX, then they have to develop CoreML's API further.

capableweb 3y ago
Not saying you're right/wrong, but argument from the other side: Apple's hardware ships with hardware specifically for ML, and a lot of it (maybe even all of the recently released?) has shared memory between GPU/CPU, meaning you could run larger models. Windows/Linux is currently stuck with having to run inference with either RAM or VRAM, limiting the options quite a bit.
jroesch 3y ago
Things are already possible on today's hardware, see https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/mlc-llm which allows many models to be run on M1/M2 Macs, WASM, iOS and more. The main limiting factor will be small enough, high quality enough models that performance is high enough ultimately this is HW limited and they will need to improve the neural engine/map more computation on to it to make the mobile exp. possible.