3 ms·
> The the question is if a llm will run with usable performance at that scale? This is the big question to have answered. Many people claim Apple can now relia
by diggan 2y ago
> The the question is if a llm will run with usable performance at that scale?
This is the big question to have answered. Many people claim Apple can now reliably be used as a ML workstation, but from the numbers I've seen from benchmarks, the models may fit in memory, but the performance for tok/sec is so slow to not feel worth it, compared to running it on NVIDIA hardware.
Although it be expensive as hell to get 512GB of VRAM with NVIDIA today, maybe moves like this from Apple could push down the prices at least a little bit.
- radlad 2y agoIt is much slower than nVidia, but for a lot of personal-use LLM scenarios, it's very workable. And it doesn't need to be anywhere near as fast considering it's really the only viable (affordable) option for private, local inference, besides building a server like this, which is no faster: https://news.ycombinator.com/item?id=42897205 https://news.ycombinator.com/item?id=42897205
- bastardoperator 2y agoIt's fast enough for me to cancel monthly AI services on a mac mini m4 max.
- staticman2 2y agoSmaller, dumber models are faster than bigger, slower ones. What model do you find fast enough and smart enough?
- Matl 2y agoNot OP but I am finding the Qwen 2.5 32b distilled with DeepSeek R1 model to be a good speed/smartness ratio on the M4 Pro Mac Mini.
- a1o 2y agoHow much RAM?
- bastardoperator 2y agoI'm running the same exact models.
- diggan 2y agoCould you maybe share a lightweight benchmark where you share the exact model (+ quantization if you're using that) + runtime + used settings and how much tokens/second you're getting? Or just like a log of the entire run with the stats, if you're using something like llama.cpp, LMDesktop or ollama? Also, would be neat if you could say what AI services you were subscribed to, there is a huge difference between paid Claude subscription and the OpenAI Pro subscription for example, both in terms of cost and the quality of responses.
- deleted 2y ago[deleted]
- fetus8 2y agoHow much RAM are you running on?
- lostmsu 2y agoHm, the AI services over 5 years cost half of m4 max minimal configuration which can barely run severely lobotomized LLaMA 70B. And they provide significantly better models.
- jamesy0ung 2y agoI presume you're using the Pro, not the Max. Anyways, what ram config, and what model are you using?
- hangonhn 2y agoDo we know if is it slower because of hardware is not as well suited for the task or is it mostly a software issue -- the code hasn't been optimized to run on Apple Silicon?
- titzer 2y agoAFAICT the neural engine has accelerators for CNNs and integer math, but not the exact tensor operations in popular LLM transformer architectures that are well-supported in GPUs.
- kridsdale1 2y agoI have to assume they’re doing something like that in the lab for 4 years from now.
- woadwarrior01 2y agoThe neural engine is perfectly capable of accelerating matmults. It's just that autoregressive decoding in single batch LLM inference is memory bandwidth constrained, so there are no performance benefits to using the ANE for LLM inference (although, there's a huge power efficiency benefit). And the only way to use the neural engine is via CoreML. Using the GPU with MLX or MPS is often easier.
- azinman2 2y agoMemory bandwidth is the issue