2 ms·
Last month I did testing on the latest gen AMD APU (7940HS @ 65W - 8C Zen4 CPU, Radeon 780M RDNA3 GPU) on how llama2-7b (4-bit quants) performed w/ a variety of
by lhl 3y ago
Last month I did testing on the latest gen AMD APU (7940HS @ 65W - 8C Zen4 CPU, Radeon 780M RDNA3 GPU) on how llama2-7b (4-bit quants) performed w/ a variety of inferencers (llama.cpp, exllama v2, mlc w/ CPU, CLBlast, ROCm, and Vulkan backends) and only llama.cpp's ROCm implementation convincingly beating CPU inferencing.
Even then, interactive (batch=1) inferencing was still barely faster. I suspect memory bandwidth (I ran w/ 2 x DDR5-5600) is the main bottleneck. Just as a point of reference, the inferencing performance is roughly in the ballpark of (but slightly lower than) my 16GB M2 MBA.
Details: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3aePczz3zlbJW-Y4/edit#gid=1041125589 https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...
- atq2119 3y agoYes. Without batching, every weight is only used once per token. On discrete GPUs, you generally need around ~20 flops/byte of memory bandwidth to be compute bound. I don't know where APUs are in that regard, but probably not entirely different. Memory bandwidth divided by model size is more or less the token rate you should expect to get from a good LLM inference implementation.
- lhl 3y ago2 channels of DDR5-5600 = 89.6GB/s theoretical max bandwidth and a Q4_0 quant takes about 6.7GB of VRAM used, which is 13.4X, so that maps out pretty well for inferencing speed (I think the divided by parameters maybe only works for 4bit in this case). Interestingly, the ROCm implementation does go a bit faster (>15 t/s so it must have some tricks up its sleeve). The Radeon 780M does 8.9 TFLOPS of FP32, 17.8 TFLOPS of FP16, so if 20 flops/byte applies, it has roughly 10X more compute than memory bandwidth for inferencing.