3 ms·

I've been writing micro memory benchmarks and have been rather surprised how hard something as simple as quantifying latency and bandwidth under multicore loads

by sliken 4y ago

I've been writing micro memory benchmarks and have been rather surprised how hard something as simple as quantifying latency and bandwidth under multicore loads can be. The memory hierarchy is getting ever more complex. Cacheline sizes, prefetch, 3 levels of cache, TLB effects, page alignments, cache associativity, etc. Also have to be careful that the compiler doesn't optimize away parts of your code. It's quite tricky to get a nice clean array size vs latency graph, doubly so when multiple cores are involved. Some of my assumptions about latency were wrong. One thing I didn't realize is it takes about half the latency to main memory to get miss through L1, L2, and L3. Also that you need to have around 2x the memory references pending to keep the memory system busy. It makes sense in retrospect, you want 16 pending memory references to keep 8 memory channels busy, otherwise a memory channel will return a cache line, and there won't be any L3 cache misses pending for that channel. Generally I like to keep a small histogram of cycle counters to make sure I'm seeing the distribution I expect, seeing an unusual distribution is key for tracking down something you didn't account for.

hinkley 4y ago
I want a viewport onto a parallel universe where we embraced NUMA more thoroughly, and instead of making processors with ever growing layers of transparent caching, we just put 16MB of working memory on each chip. Partitionable for concurrent workloads. Maybe it could never have been then, but maybe it can be now, or soon. For instance a borrow checker might be a very good way to help decide how and when to move working memory between processors and main memory. But what do you do with all of the existing C code? I think for backward compatibility you'd need to implement virtual memory at the working memory layer. It wouldn't be the fastest code, but it would work.
- sliken 4y ago
  Well the tools are there, and the hardware is there as well. Early AMD epycs had multiple chiplets, each with it's own memory controller. However that means that even on a single socket different chiplets would see different memory latencies and some apps/benchmarks people cared about handled it poorly. The newest AMD Epycs have a unified memory controller per socket, so all chiplets see the same memory latency. But now Intel entered the chiplet game, and now has one memory controller per chiplet again. There's plenty of support for NUMA support in the kernel and various language runtimes. Apps and OSs can pin processes to cores, allocate local ram, and migrate non-local ram to local or migrate processes to cores that are memory local. Generally the automatic stuff works reasonably, but numactl and various NUMA related calls let you override those decisions as needed.
  - hinkley 4y ago
    I missed the 16 bit era by dragging my feet and then leaping onto 32 bit as soon as it was viable. 64k of memory was not really enough to do useful things and it occupied so much of your design. But 4GB is a pretty big leap from 64k. There's lots of interesting things a person can do with 10, 30, 100MB of memory, without having to treat every piece of data like it sits in a perfectly flat data plane. And with 64 bit addressing you have ample, ample space to use some bits as metadata that lets you know things about the memory without dereferencing it first.
    - sliken 4y ago
      Heh, I was running DOS and Deskview, windows 3 was out of coming soon. I jumped into Linux with a 386sx-16, not even a FPU. I think it had 4MB ram, eventually upgraded to 16mb. Your ideas remind me of the Sun MAJC CPU. Claimed to be designed to run Java well, be aware of Java Objects, enabling prefetch of objects, speculative execution without side effects, and other pointer magic related to assuming JVM+Java instead of C/C++. Sounded really promising, sadly died shortly afterwards. An expert I talked with later mentioned that it was all marketing and it was just a normal CPU for it's time. Sad. Does seem like today's new languages could result in a CPU with significantly more freedom for speculation, optimization, and efficiency if it assumed a particular language runtime.
      - hinkley 4y ago
        > Your ideas remind me of the Sun MAJC CPU. Probably not a coincidence. It was memory of some Sun product data sheets that got me ruminating on this line of thought, though I don't think it was MAJC. 4MB per core, 4 cores per daughter card, if memory serves. Those product lines were probably a big reason why Java had to solve the NUMA memory problem, and then other languages just copied what they did.
  - marcosdumay 4y ago
    Hum... The hardware isn't actually there. It has the capabilities the GP wants, but it doesn't export them in a way you could tune into that NUMA machine.
- singron 4y ago
  This is basically how tile renderers work in mobile GPUs. Supercomputers also have a similar design with MPI and toroid networks although on a multi-machine scale.
- Dylan16807 4y ago
  Partitioning memory and cores is easy enough, and a bunch of Xeons have cache partitioning. It's still generally transparent to the program, but I don't think that makes it less NUMA.
- vvanders 4y ago
  We kinda had that world with the PS3 and SPE/SPUs[1]. Stuff written for them tended to run faster on more traditional hardware since you needed to vectorize all of your loads(which made the prefetcher on other hardware really happy) but it was an absolute beast to port in any code. I remember stuff from PC/X360 just running like absolute garbage(not that multicore was well handled anyway during that era) until significant refactoring was done(up to and even including full rewrites in some cases). [1] https://en.m.wikipedia.org/wiki/Cell_(processor)#Synergistic_Processing_Elements_(SPE) https://en.m.wikipedia.org/wiki/Cell_(processor)#Synergistic...
- geocar 4y ago
  > we just put 16MB of working memory on each chip. Just? The chips we've got have between 32kb and 64kb -- maybe 128kb in some really big chips, and it's been that way for forty years! Everything else has to swap or cache through messaging layers (L2, L3) with real latency because we can't "just" put a bunch of memory on a chip. I mean, I like thinking about star-trek computers too, but "just" isn't the word I would use for something like this... What would you possibly do with so much memory in a circuit? > But what do you do with all of the existing C code? I think for backward compatibility you'd need to implement virtual memory at the working memory layer. People rewrote almost everything in Java. And are doing it again with Rust. Programmers want to program, and if they can't come up with anything else to do, they'll just rewrite what someone else did. I don't think you should valuate existing C code the way its owners do, but by how their competition will valuate it; If someone can use FutureLang to outperform their business-competition who uses C, this problem will solve itself.
- jeffreygoesto 4y ago
  Reminds me of Movidius Myriad [0] or NEC IMAPCAR [1]. Other comments also mention Cell. Programmung those is so different that you need to dressing specifically for the chip and unless there is no great consolidation, they stay niche. [0] https://www.tomshardware.com/news/movidiud-myriad2-vpu-vision-processing-vr,30850.html https://www.tomshardware.com/news/movidiud-myriad2-vpu-visio... [1] https://www.nec.com/en/global/techrep/journal/g06/n05/pdf/t060507.pdf https://www.nec.com/en/global/techrep/journal/g06/n05/pdf/t0...

by sliken 4y ago

hinkley 4y ago

I want a viewport onto a parallel universe where we embraced NUMA more thoroughly, and instead of making processors with ever growing layers of transparent caching, we just put 16MB of working memory on each chip. Partitionable for concurrent workloads. Maybe it could never have been then, but maybe it can be now, or soon. For instance a borrow checker might be a very good way to help decide how and when to move working memory between processors and main memory. But what do you do with all of the existing C code? I think for backward compatibility you'd need to implement virtual memory at the working memory layer. It wouldn't be the fastest code, but it would work.

sliken 4y ago
Well the tools are there, and the hardware is there as well. Early AMD epycs had multiple chiplets, each with it's own memory controller. However that means that even on a single socket different chiplets would see different memory latencies and some apps/benchmarks people cared about handled it poorly. The newest AMD Epycs have a unified memory controller per socket, so all chiplets see the same memory latency. But now Intel entered the chiplet game, and now has one memory controller per chiplet again. There's plenty of support for NUMA support in the kernel and various language runtimes. Apps and OSs can pin processes to cores, allocate local ram, and migrate non-local ram to local or migrate processes to cores that are memory local. Generally the automatic stuff works reasonably, but numactl and various NUMA related calls let you override those decisions as needed.
- hinkley 4y ago
  I missed the 16 bit era by dragging my feet and then leaping onto 32 bit as soon as it was viable. 64k of memory was not really enough to do useful things and it occupied so much of your design. But 4GB is a pretty big leap from 64k. There's lots of interesting things a person can do with 10, 30, 100MB of memory, without having to treat every piece of data like it sits in a perfectly flat data plane. And with 64 bit addressing you have ample, ample space to use some bits as metadata that lets you know things about the memory without dereferencing it first.
  - sliken 4y ago
    Heh, I was running DOS and Deskview, windows 3 was out of coming soon. I jumped into Linux with a 386sx-16, not even a FPU. I think it had 4MB ram, eventually upgraded to 16mb. Your ideas remind me of the Sun MAJC CPU. Claimed to be designed to run Java well, be aware of Java Objects, enabling prefetch of objects, speculative execution without side effects, and other pointer magic related to assuming JVM+Java instead of C/C++. Sounded really promising, sadly died shortly afterwards. An expert I talked with later mentioned that it was all marketing and it was just a normal CPU for it's time. Sad. Does seem like today's new languages could result in a CPU with significantly more freedom for speculation, optimization, and efficiency if it assumed a particular language runtime.
    - hinkley 4y ago
      > Your ideas remind me of the Sun MAJC CPU. Probably not a coincidence. It was memory of some Sun product data sheets that got me ruminating on this line of thought, though I don't think it was MAJC. 4MB per core, 4 cores per daughter card, if memory serves. Those product lines were probably a big reason why Java had to solve the NUMA memory problem, and then other languages just copied what they did.
- marcosdumay 4y ago
  Hum... The hardware isn't actually there. It has the capabilities the GP wants, but it doesn't export them in a way you could tune into that NUMA machine.
singron 4y ago
This is basically how tile renderers work in mobile GPUs. Supercomputers also have a similar design with MPI and toroid networks although on a multi-machine scale.
Dylan16807 4y ago
Partitioning memory and cores is easy enough, and a bunch of Xeons have cache partitioning. It's still generally transparent to the program, but I don't think that makes it less NUMA.
vvanders 4y ago
We kinda had that world with the PS3 and SPE/SPUs[1]. Stuff written for them tended to run faster on more traditional hardware since you needed to vectorize all of your loads(which made the prefetcher on other hardware really happy) but it was an absolute beast to port in any code. I remember stuff from PC/X360 just running like absolute garbage(not that multicore was well handled anyway during that era) until significant refactoring was done(up to and even including full rewrites in some cases). [1] https://en.m.wikipedia.org/wiki/Cell_(processor)#Synergistic_Processing_Elements_(SPE) https://en.m.wikipedia.org/wiki/Cell_(processor)#Synergistic...
geocar 4y ago
> we just put 16MB of working memory on each chip. Just? The chips we've got have between 32kb and 64kb -- maybe 128kb in some really big chips, and it's been that way for forty years! Everything else has to swap or cache through messaging layers (L2, L3) with real latency because we can't "just" put a bunch of memory on a chip. I mean, I like thinking about star-trek computers too, but "just" isn't the word I would use for something like this... What would you possibly do with so much memory in a circuit? > But what do you do with all of the existing C code? I think for backward compatibility you'd need to implement virtual memory at the working memory layer. People rewrote almost everything in Java. And are doing it again with Rust. Programmers want to program, and if they can't come up with anything else to do, they'll just rewrite what someone else did. I don't think you should valuate existing C code the way its owners do, but by how their competition will valuate it; If someone can use FutureLang to outperform their business-competition who uses C, this problem will solve itself.
jeffreygoesto 4y ago
Reminds me of Movidius Myriad [0] or NEC IMAPCAR [1]. Other comments also mention Cell. Programmung those is so different that you need to dressing specifically for the chip and unless there is no great consolidation, they stay niche. [0] https://www.tomshardware.com/news/movidiud-myriad2-vpu-vision-processing-vr,30850.html https://www.tomshardware.com/news/movidiud-myriad2-vpu-visio... [1] https://www.nec.com/en/global/techrep/journal/g06/n05/pdf/t060507.pdf https://www.nec.com/en/global/techrep/journal/g06/n05/pdf/t0...