3 ms·

Apple predicted the rise of local LLMs, hence the M2 Ultra

2 points by behnamoh 3y ago

When Mac Studio was announced, my first question was: Why cram so much power in a device when no software needs it? I'm talking about the 192GB RAM—when was the last time you needed that? Apple was (and is) not a player in desktop gaming and it shows—the M series are amazing in terms of power use, but not comparable to dedicated nvidia GPUs, so, again, why 192GB RAM? It was until recently when I read this [0] that I realized Apple might have very well predicted the rise of local LLMs and most likely has its own in-house LLM in the works. In [0], the GG in GGML and GGUF explains that "The M2 Ultra is the absolute best personal LLM inference node that you can buy today." And remember: llama.cpp is not even taking full advantage of the modern APIs Apple announced. Still, it achieves amazing inference speeds. Imagine the day when llama.cpp goes all Apple and integrates their API... Apple well knew that local LLMs are the future, and they predicted that running a small model (up to 34B) should not require much processing power (so the M series need not be comparable to nvidia RTX GPUs) and this translates directly to much lower power consumption. But the bottleneck that Apple foresaw was the amount of graphic memory that these models are going to need. So it's no wonder that you see high RAM and low GPU power in the M series. [0]: https://github.com/ggerganov/llama.cpp/discussions/3026

smoldesu 3y ago
Is this a joke? Seriously, look at the competitors: - Google, who had been developing and distributing Tensorflow and Tensorflow-based hardware for computer vision and early transformer models before "CoreML" even existed, let alone shipped. - Facebook, who wrote and distributed Pytorch, and trained + released Llama before an Apple press release even uttered "LLM". They also ship dozens of AI-powered features on the server-side and locally on Quest. - Nvidia, who had been shipping 192gb+ systems for over a decade, as well as researching GANs, LLMs and Neural Nets and putting those features in CUDA as early as 2014. - Microsoft, who not only wrote their own CoreML-styled API (Olive/DirectML), but actually maintains a cross-platform AI toolkit with ONNX. ...and Apple, the least-vocal company WRT AI is somehow the one that predicted LLMs with the release of an iterative product update? You'd have better evidence if you backed this up with astrology or gematria. > And remember: llama.cpp is not even taking full advantage of the modern APIs Apple announced Because as-is, CoreML does not have the features required to run GGML models out-of-the-box. Suffice to say nobody is really investing in Apple for their ML potential. If you wanna catch falling knives though, be my guest.
- behnamoh 3y ago
  Is this a joke? None of the companies that you mentioned have any incentive to let users run local LLMs. And Facebook/Meta doesn't make hardware (GPU) in case you didn't know.
  - smoldesu 3y ago
    Regardless of incentive; every company I just mentioned already does more to let users run LLMs than Apple. Google wrote Tensorflow, Meta designed Pytorch, Nvidia wrote CUDA and the fundamental AI papers, and Microsoft distributes ONNX. By comparison, Apple... - Doesn't support PCIe inferencing hardware - Refuses to sign their competitor's drivers - Ships a neural engine they ignore for significant tasks - Doubled-down on the proprietary CoreML API and model format - Cut funding for OpenCL, their original Open Source GPGPU effort - Doesn't provide Open Source GPU compute primatives for their own GPU hardware - Actively ignores efforts to Open Source said hardware with better Open Source drivers - Continues converting models to their own format instead of fixing Apple hardware as a compute target ...and you mean to tell me that the M2 "predicted" LLMs? By supporting more memory? Apple was the last company out of everyone mentioned that even started working with AI. It's fine if you enjoy your Apple Silicon Mac, but it no more "predicted the rise of local LLMs" than AVX and SSE did in 1999. You made no effort to correlate the two ideas, and Apple's antagonistic role in open standards development kinda offsets whatever slight victory you can claim. Even Nvidia does more for local LLM development, and that's sad.

2 points by behnamoh 3y ago

smoldesu 3y ago

Is this a joke? Seriously, look at the competitors: - Google, who had been developing and distributing Tensorflow and Tensorflow-based hardware for computer vision and early transformer models before "CoreML" even existed, let alone shipped. - Facebook, who wrote and distributed Pytorch, and trained + released Llama before an Apple press release even uttered "LLM". They also ship dozens of AI-powered features on the server-side and locally on Quest. - Nvidia, who had been shipping 192gb+ systems for over a decade, as well as researching GANs, LLMs and Neural Nets and putting those features in CUDA as early as 2014. - Microsoft, who not only wrote their own CoreML-styled API (Olive/DirectML), but actually maintains a cross-platform AI toolkit with ONNX. ...and Apple, the least-vocal company WRT AI is somehow the one that predicted LLMs with the release of an iterative product update? You'd have better evidence if you backed this up with astrology or gematria. > And remember: llama.cpp is not even taking full advantage of the modern APIs Apple announced Because as-is, CoreML does not have the features required to run GGML models out-of-the-box. Suffice to say nobody is really investing in Apple for their ML potential. If you wanna catch falling knives though, be my guest.

behnamoh 3y ago
Is this a joke? None of the companies that you mentioned have any incentive to let users run local LLMs. And Facebook/Meta doesn't make hardware (GPU) in case you didn't know.
- smoldesu 3y ago
  Regardless of incentive; every company I just mentioned already does more to let users run LLMs than Apple. Google wrote Tensorflow, Meta designed Pytorch, Nvidia wrote CUDA and the fundamental AI papers, and Microsoft distributes ONNX. By comparison, Apple... - Doesn't support PCIe inferencing hardware - Refuses to sign their competitor's drivers - Ships a neural engine they ignore for significant tasks - Doubled-down on the proprietary CoreML API and model format - Cut funding for OpenCL, their original Open Source GPGPU effort - Doesn't provide Open Source GPU compute primatives for their own GPU hardware - Actively ignores efforts to Open Source said hardware with better Open Source drivers - Continues converting models to their own format instead of fixing Apple hardware as a compute target ...and you mean to tell me that the M2 "predicted" LLMs? By supporting more memory? Apple was the last company out of everyone mentioned that even started working with AI. It's fine if you enjoy your Apple Silicon Mac, but it no more "predicted the rise of local LLMs" than AVX and SSE did in 1999. You made no effort to correlate the two ideas, and Apple's antagonistic role in open standards development kinda offsets whatever slight victory you can claim. Even Nvidia does more for local LLM development, and that's sad.