4 ms·
The lookup table does surprisingly well because the workload is otherwise extremely cache-hostile, and it doesn't really matter if you blow up your L1 cache, no
by Tuna-Fish 2y ago
The lookup table does surprisingly well because the workload is otherwise extremely cache-hostile, and it doesn't really matter if you blow up your L1 cache, none of the data you evicted because you needed to fit the LUT was ever going to be reused anyway.
ML loads in general are streaming loads that linearly load the entire dataset for every iteration.
- pclmulqdq 2y agoOne interesting option for these big memory scans on x86 and ARM CPUs is using the non-temporal load/store instructions. Those actually bypass caching* and may help with the cache pressure of LLM workloads that just do scans. The lookup table is still probably the wrong solution even with this sort of thing. * Not quite all of it - There are still buffers to do write combining and some read caching on scans.