3 ms·

When doing pointer chasing, then it's gonna be more of a latency problem, there can be plenty of memory bandwidth available, but we don't know which memory line

When doing pointer chasing, then it's gonna be more of a latency problem, there can be plenty of memory bandwidth available, but we don't know which memory line we want to go to next, before the previous line (where the pointer resides) has been loaded. So, the CPUs spend a lot of time in "stalled backend" mode due to the big difference in CPU cycle latency vs RAM access latency. Offloading some operations closer to the RAM would be a hardware solution (like the Oracle/Sun SPARC DAX I mentioned in a separate comment). Or you could design your software to rely more on the memory throughput (scanning through columnar structures) vs memory latency (pointer chasing). Btw, even with pointer-chasing, you could optimize the application to work mostly from the CPU cache (assuming that the CPUs don't do concurrent writes into these cache lines all the time), but this would require not only different application code, but different underlying data structures too. That's pretty much what my article series is about - fancy CPU throuhgput features like SIMD would not be very helpful, if the underlying data (and memory) structures don't support their way of thinking.

vlovich123 6y ago
I’m thinking more of something like a GC mark and sweep. You don’t care about how long the mark (or even sweep) operations take as long as they finish eventually. Those operations would be the easiest to write an algorithm you could offload to run directly inside a memory controller. These would avoid GC pauses (if using a concurrent GC) while also avoiding any CPU cycles doing anything but a trivial amount of small operations. What stood out to me with this article is that the fancy “intelligent” columnar indices were actually performing worse than a naive brute traversal of the data. This is because there is a limit to your ability to predict whether a brute solution is faster or slower ahead of time than one using a fancy data structure. So a HW-offload is nice because in theory it should have less of an impact and you can run memory-intensive operations concurrently with CPU-intensive operations with no costs (+ the memory heavy operations might run faster due to living closer to the memory and thus having faster interconnects and lower latency). Additionally, such HW can take advantage of the peculiarities of how RAM works for even added benefit. Your memory allocator could mark a block as unused (in the simplest v0 of what such Hw code do). The HW can then avoid refreshing those rows reducing power usage slightly, improving latency (since refresh is a stop-the-world operation) if you can get fine-grained enough and it’s not $ prohibitive. Garbage collection would certainly be a lot more attractive of a solution for this kind of design which would drastically change the SW economics (ie Rust/C/C++ suddenly lose a whole lot of the practical perf wins they hold over other languages).

vlovich123 6y ago

I’m thinking more of something like a GC mark and sweep. You don’t care about how long the mark (or even sweep) operations take as long as they finish eventually. Those operations would be the easiest to write an algorithm you could offload to run directly inside a memory controller. These would avoid GC pauses (if using a concurrent GC) while also avoiding any CPU cycles doing anything but a trivial amount of small operations. What stood out to me with this article is that the fancy “intelligent” columnar indices were actually performing worse than a naive brute traversal of the data. This is because there is a limit to your ability to predict whether a brute solution is faster or slower ahead of time than one using a fancy data structure. So a HW-offload is nice because in theory it should have less of an impact and you can run memory-intensive operations concurrently with CPU-intensive operations with no costs (+ the memory heavy operations might run faster due to living closer to the memory and thus having faster interconnects and lower latency). Additionally, such HW can take advantage of the peculiarities of how RAM works for even added benefit. Your memory allocator could mark a block as unused (in the simplest v0 of what such Hw code do). The HW can then avoid refreshing those rows reducing power usage slightly, improving latency (since refresh is a stop-the-world operation) if you can get fine-grained enough and it’s not $ prohibitive. Garbage collection would certainly be a lot more attractive of a solution for this kind of design which would drastically change the SW economics (ie Rust/C/C++ suddenly lose a whole lot of the practical perf wins they hold over other languages).