5 ms·
Once the data exceed L3 cache, 4-bit is fastest since the dot product is memory bound: data movement from RAM becomes the bottleneck. The speed-up is up
by 0xfaded 8y ago
Once the data exceed L3 cache,
4-bit is fastest since the dot
product is memory bound: data
movement from RAM becomes the bottleneck.
The speed-up is up to 6x over
the 32-bit version.
Large linalg operations are memory bound.
I learned this the hard way hand coding an arm32 5x5 gaussian blur. When benchmarking my first version, using two passes of the separatable 5x1 and 1x5 filters, I had a facepalm moment when I realized I was really only measuring two round trips to main memory.
Operating on 16x8 pixel blocks, which fit within the register file, required most pixels to only be fetched once and almost doubled performance.
- AstralStorm 8y agoAlso true for most GPUs which are ROP limited. Even more so as register files there are big.
- vanderZwan 8y agoObligatory link to Halide, a language that is all about making hand-written scheduling easier: http://halide-lang.org/ http://halide-lang.org/
- 0xfaded 8y agoHalide is cool, but from when I last looked at it its best at optimizing for cache usage. For my example, there would have still been two round trips, but to L1 instead of something farther. I've compared using GCC intrinsics vs what I would have wanted. It doesn't really get instruction scheduling right. Things like efficiently using pipelines by grouping, say, multiplication instructions and not stalling by ensuring the right number of cycles between them. I have some specialised block matrix multiplication which basically never stall the processor. We're talking 10x speedups over optimized but general c++ code.
- Nokinside 8y agoJust imagine the speedup after somebody finally integrates vector dot product operation into DRAM silicon. Currently mixing CMOS and DRAM prosesses into same silicon is not cost effective for mass production. I think mixing of the processes and heating issues are the main roadblocks.
- kjeetgill 8y agoI've never really worked at such a fine level. I'd love to hear more about how you measured and benchmarked things and were able to draw conclusions from them. When benchmarking, I usually have a larger workload in mind. I'm usually working on and deciding between two significant algorithmic changes and I'm looking at differences in 100-1000ms and I still find the standard deviations unsettling.