7 ms·
"Light inference" insofar as inference that can actually run on consumer hardware. MPS support is lacking in many ways, i.e. BF16 support is only arriving with
by ryneandal 3y ago
"Light inference" insofar as inference that can actually run on consumer hardware.
MPS support is lacking in many ways, i.e. BF16 support is only arriving with Sonoma. One of the biggest things Nvidia has going for it is software maturity, CUDA is far beyond any alternative, particularly MPS. Many PyTorch optimizations only support Nvidia GPUs.
And the tensor core equivalents on Apple SoCs are in the ANE, which necessitates using CoreML. While unified memory bandwidth is impressive on Apple SoCs, it is still slower than recent gen Nvidia cards.
While the PPW certainly is impressive and the overall ML performance is impressive for laptop SoCs, Apple Silicon simply does not hold a candle to discrete GPU performance. You're correct, it can certainly run local inference on the memory-hungry, demanding LLMs that are en vogue currently, but you won't see the performance of a discrete GPU.
The unified memory lends itself to a cheaper, yet slower, alternative to DC-grade Nvidia cards with massive amounts of HBM, but there's a reason you don't see racks of Apple Silicon for LLM training, or even inference at scale.
I'm hopeful the software will vastly improve as Apple Silicon adoption grows, because I'd love to run some of the stuff I build for school on my MacBook, but for now I offload it to my desktop PC because the raw compute brings more value to me than the efficiency when due dates approach.
- sliken 3y agoThanks for the detailed reply, I've been reading about the tools, and tracking various forums discussing running local LLMs. ANE isn't required, just optional. However the Apple M2 MAX GPU seems to do inference decently when using code targetting Metal (Apple's GPU API). Apparently inference on the llama 65B model with a M2 max happens at 5 tokens/sec. Not an amazing perf/$ for running inference at scale, but pretty interesting for a developer to tinker with. While the M2 Max with 96GB ram is slower than a 4090 it can run larger models and I'm not expecting to be particularly performance limited for home/local inference.
- ryneandal 3y agoYeah, the performance is honestly incredible for the (comparatively) poor software libraries, limited GPU cores, and low ceiling for power. I adore my MacBook Pro and find it to be more than enough for my current needs. It's been surprisingly performant for my transformers course this summer. Apple Silicon is an even better deal for learning/tinkering with ML models if you already own one or plan to purchase one regardless. As you mentioned, the unified memory architecture is a cheaper alternative to massive/expensive Nvidia GPUs for running inference on larger models. The software is only going to improve, given the growing enthusiast cohort and Apple's push towards providing a solid ML development pipeline on MacOS.