TLDR: registers are faster to access than memory. Even warp-local on-chip memory. While CUDA looks like scalar code, GPUs are vector based, that scalar code is

TLDR: registers are faster to access than memory. Even warp-local on-chip memory. While CUDA looks like scalar code, GPUs are vector based, that scalar code is executed by 32-wide vector units. The threads running on the same units are called warp. There're little-known intrinsics in CUDA, __shfl_sync, __shfl_up_sync, __shfl_down_sync, that allow to permute data across lanes, thus exchanging data across threads of the warp. Replacing local memory access with registers is what gave the profit.