4 ms·

4-bit quantized llama gets borderline-acceptable performance on an iPad CPU[1], and there's no reason they can't further optimise CPU perf for this task. IIUC,

4-bit quantized llama gets borderline-acceptable performance on an iPad CPU[1], and there's no reason they can't further optimise CPU perf for this task. IIUC, LLM inference is mostly constrained by memory size and bandwidth (hence the aggressive quantization), rather than raw compute. [1] https://github.com/ggerganov/llama.cpp/discussions/844 https://github.com/ggerganov/llama.cpp/discussions/844

michaelt 3y ago
I guess it depends on how you're defining "great" LLM performance and "CPU performance supremacy versus Apple" Personally I would say "borderline-acceptable performance" with a 7B parameter model at 4-bit quantisation is nowhere near "great" or "supremacy". You can fit that on ~7GB of RAM, using your grandpa's 2016-era nvidia 1080. To take the ARM-CPU-LLM-performance crown from Apple you have to beat a fully equipped M2 Max, which provides 96GB of memory with 400GB/s of bandwidth. That's enough to run 70B parameter models in 4-bit quantisation, and it can go toe-to-toe with a dual-nvidia-4090 system that draws a kilowatt of power.
- brucethemoose2 3y ago
  Apple achieves that performance on the IGP. Once the context grows, even M2 Max CPU alone really starts to chug.
londons_explore 3y ago
low bit depth quantized puts GPU's at quite a disadvantage, since they can't use their massively parallel floating point units, and often spend a lot of time packing and unpacking numbers. However, minor tweaks to the design of GPU's could probably make them 5x quicker at directly processing quantized data (a 4 bit multiplier is so small you can afford to do thousands of them per clock cycle in a tiny silicon area), and I'm sure the next generation of GPU's will have that.
- reitzensteinm 3y ago
  Are there situations where GPUs are compute bottlenecked at batch size 1? For the time being at least, that's how local models will be used.
  - londons_explore 3y ago
    Everyone's using staged speculative decoding now and beam search. So effectively N=250 or so to get out a bit more performance.
  - brucethemoose2 3y ago
    At long contexts.

michaelt 3y ago

I guess it depends on how you're defining "great" LLM performance and "CPU performance supremacy versus Apple" Personally I would say "borderline-acceptable performance" with a 7B parameter model at 4-bit quantisation is nowhere near "great" or "supremacy". You can fit that on ~7GB of RAM, using your grandpa's 2016-era nvidia 1080. To take the ARM-CPU-LLM-performance crown from Apple you have to beat a fully equipped M2 Max, which provides 96GB of memory with 400GB/s of bandwidth. That's enough to run 70B parameter models in 4-bit quantisation, and it can go toe-to-toe with a dual-nvidia-4090 system that draws a kilowatt of power.

brucethemoose2 3y ago
Apple achieves that performance on the IGP. Once the context grows, even M2 Max CPU alone really starts to chug.

londons_explore 3y ago

low bit depth quantized puts GPU's at quite a disadvantage, since they can't use their massively parallel floating point units, and often spend a lot of time packing and unpacking numbers. However, minor tweaks to the design of GPU's could probably make them 5x quicker at directly processing quantized data (a 4 bit multiplier is so small you can afford to do thousands of them per clock cycle in a tiny silicon area), and I'm sure the next generation of GPU's will have that.

reitzensteinm 3y ago
Are there situations where GPUs are compute bottlenecked at batch size 1? For the time being at least, that's how local models will be used.
- londons_explore 3y ago
  Everyone's using staged speculative decoding now and beam search. So effectively N=250 or so to get out a bit more performance.
- brucethemoose2 3y ago
  At long contexts.