5 ms·
You want memory to be close to where it's used because at the speeds of high-performance ICs, the latency caused by distance is actually significant.
by tormeh 3y ago
You want memory to be close to where it's used because at the speeds of high-performance ICs, the latency caused by distance is actually significant.
- bschne 3y agobut isn't that aspect common among this, CPUs, GPUs, ...? And it feels like the whole NOC thing would add quite some overhead to moving things around. Are you saying proximity here more than offsets this vs. e.g. each core having its own cache as I think they do in a "normal" CPU? And if so, is this more true of ML inference workloads than other workloads, for some reason?
- loeg 3y agoI think the NOC approach is fundamentally similar to Intel's rings that they used for core interconnect back in the mid-2010s. It works. https://www.realworldtech.com/includes/images/articles/snbep-1.jpg https://www.realworldtech.com/includes/images/articles/snbep... https://en.wikichip.org/wiki/intel/microarchitectures/sandy_bridge_(client)#Ring_Interconnect https://en.wikichip.org/wiki/intel/microarchitectures/sandy_...
- paulmd 3y agoThey still use this today, and the ring interconnect is also the topology inside zen3/zen4 CCXs (with 8 cores). Ring is one of the simplest and best systems for >4 cores, until you get to about 8-10 cores (at which point you generally split it into multiple “tiers” like multiple CCXs etc).
- loeg 3y agoIntel switched to a mesh model: https://en.wikichip.org/wiki/intel/mesh_interconnect_architecture https://en.wikichip.org/wiki/intel/mesh_interconnect_archite...
- paulmd 3y agonot for consumer chips, which still use the same ringbus design. e-cores do have a CCX/core cluster, but the clusters themselves go on the ringbus lol
- adgjlsfhk1 3y agoI think the distinction with ML inference workloads is that you often have very little control flow, so this type of architecture lets you match layers to adjacent cores so that each operation gets it's data directly from the step before rather than from RAM.
- weebull 3y agoRight, so latency is less of an issue (because control flow is predictable) and performance becomes about bandwidth.
- imtringued 3y agoI honestly don't understand this latency obsession for LLMs. You are loading millions of parameters sequentially for each matrix. The access pattern is perfectly predictable. I just ran llama.cpp in with profiling and 99.9% of the time is spent in matrix multiplication. This shocked me, honestly, because I genuinely thought that there is going to be much more variety.