gregjm

3 ms·

1.
by gregjm 7mo ago
> I don't know JAX well enough to explain exactly why it's 3x faster than NumPy on the same matrix multiplications. JAX is basically a frontend for the XLA compiler, as you note. The secret sauce is two insights - 1) if you hav
2.
by gregjm 8mo ago
I wonder why H100 H2D and D2H unpinned memcpy bandwidth is *faster* on PCIe with vendor B than on SXM with vendor D. Is resizable BAR available on PCIe but not SXM? Or, could it be a software configuration difference? The driver API flag CU
3.
by gregjm 1y ago
TCMalloc never munmaps, instead it mmap(MAP_FIXED) within unpopulated PROT_NONE regions, and then madvise(MADV_FREE) at page granularity to reduce RSS. Perhaps a similar approach for file I/O could help to dodge the cost of munmap TLB
4.
by gregjm 2y ago
Re: serpentine traversal, this has to do with the .reuse suffix applied to register operands as mentioned in your link. We don’t really have control over it because it’s happening inside of ptxas during SASS generation, but when CUTLASS doe
5.
by gregjm 3y ago
The CUDA Runtime and Driver APIs have per-thread state, so using threads would unfortunately bypass our trick here to set the flag. Assuming you're on Linux, I might suggest creating a shared library to intercept calls to the Driver AP
6.
by gregjm 3y ago
> My so-called CPU “active” time is actually an inferred value; CUDA spins the CPU 100% constantly, even when the CPU is just waiting for the GPU The CUDA Runtime and Driver APIs allow you to use“blocking synchronization” where the CPU w
7.
by gregjm 3y ago
CUTLASS, which is NVIDIA’s C++ template library for writing matrix multiply and convolution kernels parametrized over input/output types, operators, and algorithm block sizes, theoretically supports this. But, each input of the (k, n)
8.
by gregjm 3y ago
> Interesting, anything in particular to read about it from Anandtech or Chips+Cheese or what? Or should I just read the whitepapers? (/sigh, reading primary sources, my only weakness) I haven’t had great luck with consulting non-pr
9.
by gregjm 3y ago
Personally I would lean away from radix sort once the problem size gets too large, on Turing they made the per-SM L1 cache coherent and on Ampere they made L2 cache significantly larger so getting maximum throughput requires more actively d