I was able to cheaply and measurably boost the perf of some heavily optimized deep learning inference stuff through judicious use of prefetch. CPU can only do t

I was able to cheaply and measurably boost the perf of some heavily optimized deep learning inference stuff through judicious use of prefetch. CPU can only do the job for you if it notices coalesced reads. It won’t do anything about that next buffer you could be pulling in now from a different place while the CPU is busy processing data it already has in the cache. Benchmaking on realistic workloads is crucial for this kind of work. If you can’t measure the benefit, leave it out, because what is harmless on one arch can be measurably harmful on another. As if that wasn’t bad enough, results can and do differ between compilers, too, even on the same arch.