4 ms·
Hoe much dedicated cache do these NPUs have? Because it's easy enough to saturate the memory bandwidth using the CPU for compute, never mind the GPU. Adding dar
by snovv_crash 8mo ago
Hoe much dedicated cache do these NPUs have? Because it's easy enough to saturate the memory bandwidth using the CPU for compute, never mind the GPU. Adding dark silicon for some special operations isn't going to make out memory bandwidth faster.
- bjackman 8mo agoDoes a cache help with inference workloads anyway? I don't know much about it but my mental model is that for transformers you need random access to billions of parameters.
- fc417fc802 8mo agoIt's streaming access, and no not as far as I'm aware. APUs have always been hilariously bottlenecked on memory bandwidth as soon as your task actually needed to pull in data. The only exception I know of is the PS5 because it uses GDDR instead of desktop memory.
- zigzag312 8mo agoAre we going to see more memory channels for consumer desktop at some point from AMD or Intel? Apple seems to be the only one that offers it.
- hypercube33 8mo agoI'm not sure what you mean - I think the mobile 300 series can do quad channel already for its APU at least. I'd assume it can do more but do you not need more slots beyond that?
- zigzag312 8mo agoI'm talking about desktop computers, not mobile.
- wtallis 8mo agoThere are a single-digit number of products using the AMD Strix Halo mobile parts that have a 256-bit memory bus. All other mobile x86 processors (including AMD's mainstream Mobile silicon used for these desktop processors) have the usual 128-bit bus.
- kristianp 8mo agoAm6 Socket isn't due for a couple of years. The current sTR5 used by Threadripper processors supports up to 8 channels of memory.
- zozbot234 8mo agoNPUs are more useful for prefill than decode anyway. Memory bandwidth is not the bottleneck for prefill.