3 ms·
Prefill (GEMM) is compute bound, decode (GEMV) is memory bound.
by zzzoom 6mo ago
Prefill (GEMM) is compute bound, decode (GEMV) is memory bound.
- Const-me 6mo ago> decode (GEMV) is memory bound Decode with batch size 1 is GEMV. Batching makes the decode GEMM too.