7 ms·
I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and
by coffeecoders 1y ago
I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit.
The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking.
What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML isn't great for custom runtimes and Apple hasn't given low-level developer access to the ANE afaik. And then there is memory bandwidth and dedicated SRAM issues. Hopefully Apple optimizes Core ML to map transformer workloads to the ANE.
- GeekyBear 1y ago> Hopefully Apple optimizes Core ML to map transformer workloads to the ANE. If you want to convert models to run on the ANE there are tools provided: > Convert models from TensorFlow, PyTorch, and other libraries to Core ML. https://apple.github.io/coremltools/docs-guides/index.html https://apple.github.io/coremltools/docs-guides/index.html
- ls-a 1y agoI thought Apple MLX can do that if you convert your model using it https://mlx-framework.org/ https://mlx-framework.org/
- GeekyBear 1y agoIt does indeed, and is more modern than Core ML.
- woadwarrior01 1y agoMLX does not support the ANE. https://github.com/ml-explore/mlx/issues/18 https://github.com/ml-explore/mlx/issues/18
- elpakal 1y agoYes it does. That’s just an issue with stale and incorrect information. Here are the docs https://opensource.apple.com/projects/mlx/ https://opensource.apple.com/projects/mlx/
- y1n0 1y agoNothing in that documentation says anything about the Apple Neural Engine. MLX runs on the GPU.
- jychang 1y agoNone of that uses the ANE.
- woadwarrior01 1y agoNo, it categorically doesn't. Not just that, it's CPU support is quite lacking (fp32 only). Currently, there are two ways to support the ANE: CoreML and MPSGraph.
- coffeecoders 1y agoIt is less about conversion and more about extending ANE support for transformer-style models or giving developers more control. The issue is in targeting specific hardware blocks. When you convert with coremltools, Core ML takes over and doesn't provide fine-grained control - run on GPU, CPU or ANE. Also, ANE isn't really designed with transformers in mind, so most LLM inference defaults to GPU.
- aurareturn 1y agoNeural Engine is optimized for power efficiency, not performance. Look for Apple to add matmul acceleration into the GPU instead. Thats how to truly speed up local LLMs.
- slacka 1y agoI too found that interesting that Apple's Neural Engine doesn't work with local LLMs. Seems like Apple, AMD, and Intel are missing the AI boat by not properly supporting their NPUs in llama.cpp. Any thoughts on why this is?
- bigyabai 1y agoMost NPUs are almost universally too weak to use for serious LLM inference. Most of the time you get better performance-per-watt out of GPU compute shaders, the majority of NPUs are dark silicon. Keep in mind - Nvidia has no NPU hardware because that functionality is baked-into their GPU architecture. AMD, Apple and Intel are all in this awkward NPU boat because they wanted to avoid competition with Nvidia and continue shipping simple raster designs.
- aurareturn 1y agoApple is in this NPU boat because they are optimized for mobile first. Nvidia does not optimize for mobile first. AMD and Intel were forced by Microsoft to add NPUs in order to sell “AI PCs”. Turns out the kind of AI that people want to run locally can’t run on an NPU. It’s too weak like you said. AMD and Intel both have matmul acceleration directly in their GPUs. Only Apple does not.
- bigyabai 1y agoNvidia's approach works just fine on mobile. Devices like the Switch have complex GPGPU pipelines and don't compromise whatsoever on power efficiency. Nonetheless, Apple's architecture on mobile doesn't have to define how they approach laptops, destops and datacenters. If the mobile-first approach is limiting their addressable market, then maybe Tim's obsessing over the wrong audience?
- aurareturn 1y agoMacBooks benefit from mobile optimization. Apple just needs to add matmul hardware acceleration into their GPUs.
- wslh 1y agoI find surprising that you can also do that from the browser (e.g. WebLLM). I imagine that in the near future we will run these engines locally for many use cases, instead of via APIs.
- ru552 1y agoYou're better served using Apple's MLX if you want to run models locally.
- jondwillis 1y agohttps://github.com/Anemll/Anemll https://github.com/Anemll/Anemll
- ai-christianson 1y agoI can run GLM 4.5 Air and gpt-oss-120b both very reasonably. GPT OSS has particularly good latency. I'm on a 128GB M4 macbook. This is "powerful" today, but it will be old news in a few years. These models are just about getting as good as the frontier models.
- giancarlostoro 1y agoI feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too safe. While Tim Cook is good as a COO, he's still running Apple as a COO. They need a man of vision, not a COO at the helm.
- jbs789 1y agoSounds like you’ve got a solid handle on things - go do it!
- giancarlostoro 1y agoGive me a majority share in AAPL if that's what you want ;)
- jbs789 1y agoThere are a things we can all do in our own lives, not necessarily running Apple. I for one am grateful not to be in the public spotlight running Apple! Everyone has opinions. It’s what you do with them that counts.
- ako 1y agoThey have local LLMs, apple foundation models: https://developer.apple.com/documentation/FoundationModels https://developer.apple.com/documentation/FoundationModels
- wer232essf 1y ago[flagged]
- o11c 1y agoIt's useless to mention number of parameters without also mentioning quantization, and to a lesser-but-still-significant extent context size, which determine how much RAM is needed. "It will run" is a different thing than "it will run without swapping or otherwise hitting a slow storage access path". That makes a speed difference of multiple orders of magnitude. This is one thing Ollama is good for. Possibly the only thing, if you listen to some of its competitors. But the choice of runner does nothing to avoid the fact that all LLMs are just toys.
- zackmorris 1y agoDon't get me started. Many new computers come with an NPU of some kind, which is superfluous to a GPU. But what's really going on is that we never got the highly multicore and distributed computers that could have started going mainstream in the 1980s, and certainly by the late 1990s when high-speed internet hit. So single-threaded performance is about the same now as 20 years ago. Meanwhile video cards have gotten exponentially more powerful and affordable, but without the virtual memory and virtualization capabilities of CPUs, so we're seeing ridiculous artificial limitations like not being able to run certain LLMs because the hardware "isn't powerful enough", rather than just having a slower experience or borrowing the PC in the next room for more computing power. To go to the incredible lengths that Apple went to in designing the M1, not just wrt hardware but in adding yet another layer of software emulation since the 68000 days, without actually bringing multicore with local memories to the level that today's VLSI design rules could allow, is laughable for me. If it wasn't so tragic. It's hard for me to live and work in a tech status quo so far removed from what I had envisioned growing up. We're practically at AGI, but also mired in ensh@ttification. Reflected in politics too. We'll have the first trillionaire before we solve world hunger, and I'm bracing for Skynet/Ultron before we have C3P0/JARVIS.
- zozbot234 1y agoFrom reverse engineered information (in the context of Asahi Linux, which can have raw hardware access to the ANE) it seems that the M1/M2 Apple Neural Engine provides exclusively for statically scheduled MADD's of INT8 or FP16 values.[0] This wastes a lot of memory bandwidth on padding in the context of newer local models which generally are more heavily quantized. (That is, when in-memory model values must be padded to FP16/INT8 this slashes your effective use of memory bandwidth, which is what determines token generation speed. GPU compute doesn't have that issue; one can simply de-quantize/pad the input in fast local registers to feed the matrix compute units, so memory bandwidth is used efficiently.) The NPU/ANE is still potentially useful for lowering power use in the context of prompt pre-processing, which is limited by raw compute as opposed to the memory bandwidth bound of token generation. (Lower power usage in this context will save on battery and may help performance by avoiding power/thermal throttling, especially on passively-cooled laptops. So this is definitely worth going for.) [0] Some historical information about bare-metal use of the ANE is available from the Whisper.cpp pull req: https://github.com/ggml-org/whisper.cpp/pull/1021 https://github.com/ggml-org/whisper.cpp/pull/1021 Even older information at: https://github.com/eiln/ane/tree/33a61249d773f8f50c02ab0b9fe5b2db58346751/docs https://github.com/eiln/ane/tree/33a61249d773f8f50c02ab0b9fe... . More extensive information at https://github.com/tinygrad/tinygrad/tree/master/extra/accel/ane https://github.com/tinygrad/tinygrad/tree/master/extra/accel... (from the Tinygrad folks) seems to basically confirm the above. (The jury is still out for M3/M4 which currently have no Asahi support - thus, no current prospects for driving the ANE bare-metal. Note however that the M3/Pro/Max ANE reported performance numbers are quite close to the M2 version, so there may not be a real improvement there either. M3 Ultra and especially the M4 series may be a different story.)
- daemonologist 1y agoONNX Runtime purports to support CoreML: https://onnxruntime.ai/docs/execution-providers/CoreML-ExecutionProvider.html https://onnxruntime.ai/docs/execution-providers/CoreML-Execu... , which gives a decent amount of compatibility for inference. I have no idea to what extent workloads actually end up on the ANE though. (Unfortunately ONNX doesn't support Vulkan, which limits it on other platforms. It's always something...)
- witnessme 1y agoDon't try 12-20B on 16GB. You should stick with 4-8B instead. You'll get way too slow tps and marginal perf improvements going higher on a 16GB machine.