3 ms·

All this stuff about warps and streaming multi-processors — is this still the best mental model for targeting modern GPUs with ML workloads? It seems cleaner to

All this stuff about warps and streaming multi-processors — is this still the best mental model for targeting modern GPUs with ML workloads? It seems cleaner to think in terms of GEMM, kernels, etc.

cavisne 3y ago
You don’t have to go to that level of detail but it is still relevant. For example FlashAttention is built on optimizing for this level of detail.
- TechnicolorByte 3y ago
  Any specifics you can describe or somewhere to read about this?
  - WithinReason 3y ago
    Just google Flash Attention 2, first hit is the paper.
tovej 3y ago
It is. A kernel is executed by a set of warps (or waves, AMD), the warps/waves define the SIMD-group that execute an instruction. Memory accesses that feed these SIMD instructions are coalesced into one operation, and therefore the optimal memory layout requires thinking in terms of warps. Kernels are executed at some blocksize, and each block is executed by a SM. The SM partitions each have a limited set of resources, and therefore the number of warps (or waves, AMD) that execute simultaneously should be tuned according to the resources available (registers, shared memory, etc.). The keyword to search for here is occupancy, and related topics include register pressure/spilling, shared memory and L1 cache-size. All extremely relevant.
Dessesaf 3y ago
Without considering how memory is accessed (coalescing all accesses in a warp), your memory accesses will be up to the warp size slower (32x for NVIDIA). So basically, you can't write an even decently efficient kernel without considering this. And if you don't consider your available shared memory, that number will be much larger. And these aren't advanced concepts, it's just the fundamental programming model for GPUs. It's on the level of "Access memory in predictable patterns, ideally sequentially" for CPUs. Everyone knows now that CPUs like arrays and sequential access. And GPUs like interleaved acccess, ideally in sizes no larger than your shared memory.

cavisne 3y ago

You don’t have to go to that level of detail but it is still relevant. For example FlashAttention is built on optimizing for this level of detail.

TechnicolorByte 3y ago
Any specifics you can describe or somewhere to read about this?
- WithinReason 3y ago
  Just google Flash Attention 2, first hit is the paper.

tovej 3y ago

It is. A kernel is executed by a set of warps (or waves, AMD), the warps/waves define the SIMD-group that execute an instruction. Memory accesses that feed these SIMD instructions are coalesced into one operation, and therefore the optimal memory layout requires thinking in terms of warps. Kernels are executed at some blocksize, and each block is executed by a SM. The SM partitions each have a limited set of resources, and therefore the number of warps (or waves, AMD) that execute simultaneously should be tuned according to the resources available (registers, shared memory, etc.). The keyword to search for here is occupancy, and related topics include register pressure/spilling, shared memory and L1 cache-size. All extremely relevant.

Dessesaf 3y ago

Without considering how memory is accessed (coalescing all accesses in a warp), your memory accesses will be up to the warp size slower (32x for NVIDIA). So basically, you can't write an even decently efficient kernel without considering this. And if you don't consider your available shared memory, that number will be much larger. And these aren't advanced concepts, it's just the fundamental programming model for GPUs. It's on the level of "Access memory in predictable patterns, ideally sequentially" for CPUs. Everyone knows now that CPUs like arrays and sequential access. And GPUs like interleaved acccess, ideally in sizes no larger than your shared memory.