3 ms·

Except in order memory access is what modern CPUs aim for (and... basically every computation device be it CPU or GPU for the last 30+ years) Strided access ha

Except in order memory access is what modern CPUs aim for (and... basically every computation device be it CPU or GPU for the last 30+ years) Strided access has always been slow, on CPUs, on GPUs, on everything, for the last 30 years. Its a known issue. "Fixing" strided memory means breaking other things (such as in-order memory access). And that's just not worth it. ------- This access pattern (strided memory access) leads to channel conflicts, bank conflicts, and other uneven memory accesses in GPU space. This entire behavior around associative caches (CPUs) or channel/bank conflicts (GPUs) is the convention, and has been for decades. I don't see any good reason to change. If it really is a problem for your code, then you can layout the memory differently in your code. (Ex: GPU programmers layout their structures to be non-power of 2, with innocuous padding in their structures to help mitigate this channel conflict problem) ------ At this point, you're gonna end up breaking all the optimized code for the last 30 years to fix slow code (that always has been slow, and therefore probably doesn't need to be optimized). That's just a bad move.

Sirened 4y ago
> Except in order memory access is what modern CPUs aim for > Strided access has always been slow, on CPUs, on GPUs, on everything Yes and no. It's not as if hardware engineers completely stuck our heads in the sand and just let the performance be shitty, they've actually developed very sophisticated prefetchers (in this case, literally called a "stride prefetcher"). Many modern stride prefetchers can detect and launch concurrent lines faults at strides of over a megabyte. Here, we're explicitly exploiting that line faults _do not_ need to happen in order thanks to memory ordering rules provided by the ISA. While you will still get crumby cache utilization due to hammering a single set, if you are legitimately just going once through a loop and operating on single items (i.e. you're streaming and not returning to items once accessed like in the article), the prefetcher will make a huge difference and get you a decent hit rate.

Sirened 4y ago

> Except in order memory access is what modern CPUs aim for > Strided access has always been slow, on CPUs, on GPUs, on everything Yes and no. It's not as if hardware engineers completely stuck our heads in the sand and just let the performance be shitty, they've actually developed very sophisticated prefetchers (in this case, literally called a "stride prefetcher"). Many modern stride prefetchers can detect and launch concurrent lines faults at strides of over a megabyte. Here, we're explicitly exploiting that line faults _do not_ need to happen in order thanks to memory ordering rules provided by the ISA. While you will still get crumby cache utilization due to hammering a single set, if you are legitimately just going once through a loop and operating on single items (i.e. you're streaming and not returning to items once accessed like in the article), the prefetcher will make a huge difference and get you a decent hit rate.