2 ms·
The bottleneck would most certainly be memory, as you'll quickly overwhelm the on-die cache, without careful optimization. That said, I think AMD's chiplet str
by bb88 3y ago
The bottleneck would most certainly be memory, as you'll quickly overwhelm the on-die cache, without careful optimization.
That said, I think AMD's chiplet strategy might come into play. I could see AMD release a 4 core 8 thread processor with increased on die cache and other chiplets being neural compute units.
- brucethemoose2 3y agoPeople keep reiterating this, but in practice one needs compute and bandwidth, especially outside of tiny context test prompts. On my 4900HS, mlc-llm vulkan is far faster than CPU inference on the same memory bus, with less cache, which wouldn't be the case if it was bandwidth/cache bound (since the CPU has far more cache as well). My 7800X3D has 96MB of L3 and a golden-bin DDR5 overclock, but its absolutely dreadful for inference.
- bb88 3y agoI don't disagree here, but the new chips here have special neural compute units, and specifically he's talking about models larger than 24GB.