3 ms·

KV-cache based LLM inference is normally significantly memory bound on the matrix-vector multiply. This is (part of) why the quantization-based approaches are s

KV-cache based LLM inference is normally significantly memory bound on the matrix-vector multiply. This is (part of) why the quantization-based approaches are so popular.

twoodfin 2y ago
It's memory bound and single threaded?
- rini17 2y ago
  I found it memory bound so that it was fastest with 6 threads on my 8 thread xeon.
  - twoodfin 2y ago
    Right: So this trick probably doesn’t help much. You only need prefetch when there isn’t enough fetch otherwise.