4 ms·
This is huge for AI / ML at least for inference. Apple chips are among the most efficient out there for that sort of thing, the only downside is the lack of cu
by wing-_-nuts 2y ago
This is huge for AI / ML at least for inference. Apple chips are among the most efficient out there for that sort of thing, the only downside is the lack of cuda
- talldayo 2y ago> Apple chips are among the most efficient out there for that sort of thing Not really? Apple is efficient because they ship moderately large GPUs manufactured on TSMC hardware. Their NPU hardware is more or less entirely ignored and their GPUs are using the same shader-based compute that Intel and AMD rely on. It's not efficient because Apple does anything different with their hardware like Nvidia does, it's efficient because they're simply using denser silicon than most opponents. Apple does make efficient chips, but AI is so much of an afterthought that I wouldn't consider them any more efficient than Intel or AMD.
- nextos 2y agoFor inference, Apple chips are great due to a high memory bandwidth. Mac Studio is a popular choice in the local Llama community for this particular reason. It's a cost effective option if you need a lot of memory plus a high bandwidth. The downside is poor training performance and Metal being a less polished software stack compared to CUDA. I wonder if a little cluster of Mac Minis is a good option for running concurrent LLM agents, or a single Mac Studio is still preferable?
- angoragoats 2y agoThe memory bandwidth on Apple silicon is only sometimes comparable to, and in many cases worse than, that of a GPU. For example, an nVidia RTX 4060 Ti 16GB GPU (not a high-end card by any means) has memory bandwidth of 288GiB/sec, which is more than double that of the M4 CPU. On the higher end, building a machine with 6 to 8 24GB GPUs such as RTX 3090s would be comparable in cost (as well as available memory) to a high-end Mac Studio, and would be at least an order of magnitude faster at inference. Yes, it's going to use an order of magnitude more power as well, but what you probably should care about here is W/token which is in the same ballpark. Apple silicon is a reasonable solution for inference only if you need the most amount of memory possible, you don't care about absolute performance, and you're unwilling to deal with a multi-GPU setup.
- Y-bar 2y agoNote that they said the _Mac Studio_ which in the M2 model has between 400GB/s and 800GB/s memory bandwidth. https://www.apple.com/mac-studio/specs/ https://www.apple.com/mac-studio/specs/ Edit: since my reply you have edited your comment to mention the Studio, but the fact remains that the M2 Max has at least ~40% greater bandwidth than the number you quoted as an example.
- angoragoats 2y agoYeah, sorry, I realized that as well so I edited my post to add a higher end example with multiple 3090s or similar cards. A single 3090 has just under 1TiB/sec of memory bandwidth. One more edit: I'd also like to point out that memory bandwidth is important, but not sufficient for fast inference. My entire point here is that Apple silicon does have high memory bandwidth for sure, but for inference it's very much held back by the relative slowness of the GPU compared with dedicated nVidia/AMD cards.
- int_19h 2y agoIt's still "fast enough" for even 120b models in practice, and you don't need to muck around with building a multi-GPU rig (and figuring out how to e.g. cool it properly). It's definitely not what you'd want for your data center, but for home tinkering it has a very clear niche.
- angoragoats 2y ago> It's still "fast enough" for even 120b models in practice Is it? This is very subjective. The Mac Studio would not be "fast enough" for me on even a 70b model, not necessarily because its output is slow, but because the prompt evaluation speed is quite bad. See [0] for example numbers; on Llama 3 70B at Q4_K_M quantization, it takes an M2 Ultra with 192GB about 8.5 seconds just to evaluate a 1024-token prompt. A machine with 6 3090s (which would likely come in cheaper than the Mac Studio) is over 6 times faster at prompt parsing. A 120b model is likely going to be something like 1.5-2x slower at prompt evaluation, rendering it pretty much unusable (again, for me). [0] https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...
- alwayslikethis 2y agoDoes it use the GPU? I was under the impression that it uses the CPU. It's only faster because of the massive memory bandwidth compared to DDR4/5
- astrange 2y agoThe AI features use all three of NPU ("ANE"), GPU, CPU, mostly depending on model size. https://machinelearning.apple.com/research/neural-engine-transformers https://machinelearning.apple.com/research/neural-engine-tra...
- DrBenCarson 2y agoFrankly you’re very wrong. NPUs and GPUs aside, 16gb of GPU memory is very rare in consumer hardware
- astrange 2y agoYou can't use all 16GB because it's unified, so it's shared with the system, SSD controller etc. You can use something like 12-14GB though.
- DrBenCarson 2y agoSure, still incredibly rare in a $600 device
- angoragoats 2y agoI'm not sure what you mean. RTX 4060 Ti/4070 Ti Super/3090/4090 cards can be easily purchased at any major electronics store in person or online and have 16GB or 24GB depending on model. Once you get up to 32GB, your point would stand, but 16-24GB GPUs are common.
- DrBenCarson 2y agoYes but the average user is not purchasing those, let alone putting together a system with one for $600
- angoragoats 2y agoYou said nothing about price in your initial comment, and the cards I listed are some of the most popular GPUs of the last several years.
- burnerthrow008 2y agoLack of Cuda is not a problem if for most ML frameworks. For example, in PyTorch you just tell it to use the “mps” (metal performance shaders) device instead of the “cuda” device.
- xattt 2y agoCuda Apple license it from nVidia?
- rob74 2y agoCude, er, cute, but... no.
- ojhughes 2y agoI tried training some models using tensorflow-metal a year ago and I was quite disappointed. Using a relu activation function led to very poor accuracy [0] and training time was an order of magnitude slower than just using the free tier of Google Colab [0] https://github.com/keras-team/tf-keras/issues/140 https://github.com/keras-team/tf-keras/issues/140
- throwaway314155 2y agoThat simply isn't true in practice. Maybe for inference, but even then you're running up against common CUDA kernels such as FlashAttention which will be far from plug and play with PyTorch.
- rowanG077 2y agoI consider that a plus. Maybe the AI community wil start to wake up and realize that going all in on cuda is ridiculously stupid.
- kridsdale3 2y agoNot if you're an NVDA shareholder!
- wing-_-nuts 2y agoTo be totally honest, there's enough money in the ML / AI / LLM space now that I fully expect some companies to put forward alternative cards specifically for that purpose. Why google does not sell their TPU to consumer and datacenter instead of just letting you rent is beyond me.
- fennecbutt 2y agoCeiling is too low on alternative hardware for that. Until Apple can bang something out as good as an h100, it's no competition. Cuda thrives bc of the hardware offering too.
- whartung 2y agoSo, do you think that when the Mac Studio gets upgraded, it will also come with less max RAM, but be unified? Is the whole "unified" RAM a reason that the iMac and Mini are capped at 32G?
- transitorykris 2y agoThe Mac Studio has always had unified memory
- kridsdale3 2y agoAll "Apple Silicon" products, going back to the first one, which was the iPhone 4.
- angoragoats 2y agoFun fact: any PC with integrated graphics has also had unified memory (yep, including Intel Macs), for at least the past decade!
- sliken 2y agoTrue, but PCs are 128 bits wide, apple lets you upgrade to 256 bits wide (M4 pro), 512 bits wide (m3 Max) or 1024 bits wide (M2 Ultra). Unified memory is much more useful when you can get more bandwidth to it.
- fulafel 2y agoYep, there's no performance x86 CPUs on the market with ambitious GPUs, only laptop chips. Games are optimized for discrete GPUs, Apple didn't have that software inertia to deal with.
- sliken 2y agoSort of, obviously quite a few games are optimized for the PS5 and Xbox series X. GPU cores are generally identical between the iGPUs and the discrete GPUs. Adding a PCIe bus (high latency and low bandwidth) and having a separate memory pool doesn't create new opportunities for optimization. On the other hand having unified memory creates optimization opportunities, but even just making memcpy a noop can be useful as well.