3 ms·

You need perf measurements to understand the impact. It sounds like you're saying that it's a massive time stick to fix the problems, so it had better be a meas

You need perf measurements to understand the impact. It sounds like you're saying that it's a massive time stick to fix the problems, so it had better be a measurable impact to understand the ROI. I've definitely seen careful memory management make a huge difference in custom RNN code, so don't doubt it can make a difference, but I expect it matters mainly when it's part of a very tight loop.

jandrewrogers 3y ago
Unlike some other optimizations, the effects of locality optimization are usually not local. The reduction in cache and bandwidth pressure makes everything run better. These are common optimizations in e.g. databases, even shaving a few bytes off of key data structures can result in system-wide gains of 5-10% performance just from the cache pressure reduction resulting from a seemingly innocuous reduction in data size. You don’t need to measure it to know it will have impact, the relationship between reducing cache pressure and performance is pretty direct and repeatable. Spatial locality is straightforward, temporal locality is a bigger engineering commitment. A canonical example of locality optimization is thread-per-core architectures versus classic multi-threading architectures. By random luck, I was around when thread-per-core architectures were first developed to improve efficiency on many-core processors (on supercomputers). Instant 10x performance improvement on the same hardware. By comparison, aggressive vectorization with AVX-512 might buy you 3x improvement. Locality optimizations are the highest leverage optimization we can do today. It does require disciplined software architecture. There are limits to locality optimization in real systems. If you are doing a good job of it, you frequently run out of memory bandwidth at some point. That’s a pretty good outcome.

jandrewrogers 3y ago

Unlike some other optimizations, the effects of locality optimization are usually not local. The reduction in cache and bandwidth pressure makes everything run better. These are common optimizations in e.g. databases, even shaving a few bytes off of key data structures can result in system-wide gains of 5-10% performance just from the cache pressure reduction resulting from a seemingly innocuous reduction in data size. You don’t need to measure it to know it will have impact, the relationship between reducing cache pressure and performance is pretty direct and repeatable. Spatial locality is straightforward, temporal locality is a bigger engineering commitment. A canonical example of locality optimization is thread-per-core architectures versus classic multi-threading architectures. By random luck, I was around when thread-per-core architectures were first developed to improve efficiency on many-core processors (on supercomputers). Instant 10x performance improvement on the same hardware. By comparison, aggressive vectorization with AVX-512 might buy you 3x improvement. Locality optimizations are the highest leverage optimization we can do today. It does require disciplined software architecture. There are limits to locality optimization in real systems. If you are doing a good job of it, you frequently run out of memory bandwidth at some point. That’s a pretty good outcome.