3 ms·
> Arm also believes that Blackhawk will provide “great” LLM performance. I will assume that this has to do with big CPU IPC performance improvements as Arm says
by brucethemoose2 3y ago
> Arm also believes that Blackhawk will provide “great” LLM performance. I will assume that this has to do with big CPU IPC performance improvements as Arm says that its Cortex CPU is the #1 AI target for developers. I didn’t run the developer or app survey myself, but this makes sense to me as most AI inference in the datacenter is run on the CPU. The NPU and GPU can be an efficient way to run AI, but a CPU is the easiest and most pervasive way, which is why developers target it. A higher-performing CPU obviously helps here, but as the world moves increasingly to smaller language models, Arm’s platform with higher-performing CPU and GPU combined with its tightly integrated ML libraries and frameworks will likely result in a more efficient experience on devices.
I feel like they are conflating 2 unrelated things?
There is "CPU inference" which is indeed pervasive and huge, and there is "generative ai llm inference" which is absolutely anemic on CPU, especially outside of tiny-context toy model demos, in spite of the popularity of llama.cpp and other cpu platforms
.
I would hope smartphone users aren't running anything but extremely tiny llms on CPU. The IGP is not theoretically, but currently massively more efficient than any 2024 CPU could be.
- adrian17 3y ago> in spite of the popularity of llama.cpp and other cpu platforms Just to clarify: llama.cpp has also added some GPU acceleration over time.
- brucethemoose2 3y agoYes, but not anything usable on IGPs.
- Retr0id 3y ago4-bit quantized llama gets borderline-acceptable performance on an iPad CPU[1], and there's no reason they can't further optimise CPU perf for this task. IIUC, LLM inference is mostly constrained by memory size and bandwidth (hence the aggressive quantization), rather than raw compute. [1] https://github.com/ggerganov/llama.cpp/discussions/844 https://github.com/ggerganov/llama.cpp/discussions/844
- michaelt 3y agoI guess it depends on how you're defining "great" LLM performance and "CPU performance supremacy versus Apple" Personally I would say "borderline-acceptable performance" with a 7B parameter model at 4-bit quantisation is nowhere near "great" or "supremacy". You can fit that on ~7GB of RAM, using your grandpa's 2016-era nvidia 1080. To take the ARM-CPU-LLM-performance crown from Apple you have to beat a fully equipped M2 Max, which provides 96GB of memory with 400GB/s of bandwidth. That's enough to run 70B parameter models in 4-bit quantisation, and it can go toe-to-toe with a dual-nvidia-4090 system that draws a kilowatt of power.
- brucethemoose2 3y agoApple achieves that performance on the IGP. Once the context grows, even M2 Max CPU alone really starts to chug.
- londons_explore 3y agolow bit depth quantized puts GPU's at quite a disadvantage, since they can't use their massively parallel floating point units, and often spend a lot of time packing and unpacking numbers. However, minor tweaks to the design of GPU's could probably make them 5x quicker at directly processing quantized data (a 4 bit multiplier is so small you can afford to do thousands of them per clock cycle in a tiny silicon area), and I'm sure the next generation of GPU's will have that.
- reitzensteinm 3y agoAre there situations where GPUs are compute bottlenecked at batch size 1? For the time being at least, that's how local models will be used.
- londons_explore 3y agoEveryone's using staged speculative decoding now and beam search. So effectively N=250 or so to get out a bit more performance.