People keep reiterating this, but in practice one needs compute and bandwidth, especially outside of tiny context test prompts. On my 4900HS, mlc-llm vulkan is

People keep reiterating this, but in practice one needs compute and bandwidth, especially outside of tiny context test prompts. On my 4900HS, mlc-llm vulkan is far faster than CPU inference on the same memory bus, with less cache, which wouldn't be the case if it was bandwidth/cache bound (since the CPU has far more cache as well). My 7800X3D has 96MB of L3 and a golden-bin DDR5 overclock, but its absolutely dreadful for inference.