4 ms·
I think the point is that this still eats into CPU <-> memory bandwidth. Offloading could let the CPU use that memory for better purposes, especially since the
by nomel 6y ago
I think the point is that this still eats into CPU <-> memory bandwidth. Offloading could let the CPU use that memory for better purposes, especially since the stall is from memory access anyways.
- geocar 6y agoLinked-list chasing (as in vlovich123's example) isn't bandwidth-limited as much as it is latency: SMT helps here because you're able to enqueue multiple wait-states "simultaneously".
- jeffbee 6y agoAnything you can do with SMT you can do without SMT using ILP instead. SMT does not grant a CPU extra cache-filling resources.
- geocar 6y agoSMT is easier to program. Walking a single linked-list is nothing; A garbage collector walks lots of them that tend to fan out for a bit.
- jeffbee 6y agoSMT gives you some probability that thread X and thread Y are under-using the resources of the CPU but if you lose that bet you get antagonism. On Intel CPUs in particular there are only 2 slots for filling cache lines, and filling them randomly from main memory takes many cycles, so two threads can easily get starved. If thread X is chasing a linked list and thread Y is compressing an MPEG then it's brilliant.
- gpsar 6y agoSoftware SMT is even better for concurrent traversals as you are not limited by the number of hw contexts https://www.linkedin.com/pulse/dont-stall-multitask-georgios-psaropoulos https://www.linkedin.com/pulse/dont-stall-multitask-georgios...
- dragontamer 6y agoDepends on the implementation of SMT. I know that POWER9 for example, reserves the superslices for different threads as you go from SMT1 (1-thread per core) up to SMT4 (4-threads per core). Thread#0 and Thread#1 get Slice0, while Thread#2 and Thread#3 get Slice1. -------- For an even more extreme example, Bulldozer's implementation of SMT / Cores / whatever you wanna call it... the L1 cache was split between the two cores. So you'd rather have 2-threads doing two different things (even if they shared the same decoder and significantly shared the same resources), because you'd "magically" get access to more L1 cache. Even on modern Skylake / Zen systems, some SMT resources are locked to one thread or the other. So you do in fact get more resources from SMT. On the other hand: you're absolutely right in that pointer-chasing can be done in parallel in a single thread due to ILP (and that ILP is probably even preferable to using more threads). Still, the fact that SMT systems are implemented so differently and so weirdly at times... it means that we can't really make general rules about SMT systems.
- jeffbee 6y agoYeah, thanks for adding details. SMT on Skylake and later is quirky, people don't appreciate that the CPU replicates some things, partitions others, and fights over some. The static partitioning in particular means that if you try to disable HT by taking all of the odd core numbers offline[+] you just wasted half of your L1 cache and iTLB entries. +: I know this sounds like an obviously-wrong approach to the problem, but I came across it at a large, well-known, public company.
- vlovich123 6y agoI agree it’s not bandwidth limited. But the CPU is stuck waiting around for the results of that operation. So offloading it to dedicated Hw let’s you do an important background task without needing the OS and CPU to spend any wall-clock time on it. Same reason you have DMAs and other coprocessors/accelerators in the first place. A good such solution will increase effective memory bandwidth (offloading heavy background-worthy tasks for the coprocessor) but also latency-sensitive tasks (eg walking a large set of linked lists, you don’t need to pull in a 256 byte cache line at a time and can instead pull the 8 bytes for the pointer). I don’t pretend that this is easy or an unbridled obvious idea. I recognize there are smarter people than me who have considered this (and from the posts clearly is being explored commercially). I am attracted to the idea on first principles that the CPU is served well by specialized coprocessors that have a performance profile better suited for some specific subset of operations. For example, memcpy is the easiest one. We still don’t have a simple (good) memcpy instruction/accelerator even though we know that an insanely large amount of work the CPU ends up doing is copying memory around in some way or another (yes Intel has one that isn’t useful/good performing).
- geocar 6y ago> I am attracted to the idea on first principles that the CPU is served well by specialized coprocessors that have a performance profile better suited for some specific subset of operations. It certainly sounds promising, that's for sure, e.g. https://link.springer.com/content/pdf/10.1007%2Fs00778-019-00581-w.pdf https://link.springer.com/content/pdf/10.1007%2Fs00778-019-0...
- tanelpoder 6y agoWhen doing pointer chasing, then it's gonna be more of a latency problem, there can be plenty of memory bandwidth available, but we don't know which memory line we want to go to next, before the previous line (where the pointer resides) has been loaded. So, the CPUs spend a lot of time in "stalled backend" mode due to the big difference in CPU cycle latency vs RAM access latency. Offloading some operations closer to the RAM would be a hardware solution (like the Oracle/Sun SPARC DAX I mentioned in a separate comment). Or you could design your software to rely more on the memory throughput (scanning through columnar structures) vs memory latency (pointer chasing). Btw, even with pointer-chasing, you could optimize the application to work mostly from the CPU cache (assuming that the CPUs don't do concurrent writes into these cache lines all the time), but this would require not only different application code, but different underlying data structures too. That's pretty much what my article series is about - fancy CPU throuhgput features like SIMD would not be very helpful, if the underlying data (and memory) structures don't support their way of thinking.
- vlovich123 6y agoI’m thinking more of something like a GC mark and sweep. You don’t care about how long the mark (or even sweep) operations take as long as they finish eventually. Those operations would be the easiest to write an algorithm you could offload to run directly inside a memory controller. These would avoid GC pauses (if using a concurrent GC) while also avoiding any CPU cycles doing anything but a trivial amount of small operations. What stood out to me with this article is that the fancy “intelligent” columnar indices were actually performing worse than a naive brute traversal of the data. This is because there is a limit to your ability to predict whether a brute solution is faster or slower ahead of time than one using a fancy data structure. So a HW-offload is nice because in theory it should have less of an impact and you can run memory-intensive operations concurrently with CPU-intensive operations with no costs (+ the memory heavy operations might run faster due to living closer to the memory and thus having faster interconnects and lower latency). Additionally, such HW can take advantage of the peculiarities of how RAM works for even added benefit. Your memory allocator could mark a block as unused (in the simplest v0 of what such Hw code do). The HW can then avoid refreshing those rows reducing power usage slightly, improving latency (since refresh is a stop-the-world operation) if you can get fine-grained enough and it’s not $ prohibitive. Garbage collection would certainly be a lot more attractive of a solution for this kind of design which would drastically change the SW economics (ie Rust/C/C++ suddenly lose a whole lot of the practical perf wins they hold over other languages).