3 ms·

I don't think this really helps: It's a trick to drive maximum possible memory bandwidth in service of a single executing thread which can process data as fast

I don't think this really helps: It's a trick to drive maximum possible memory bandwidth in service of a single executing thread which can process data as fast as it's being delivered. A parallel matrix multiply running across every core shouldn't have any trouble maximizing memory bandwidth utilization.

owlbite 2y ago
KV-cache based LLM inference is normally significantly memory bound on the matrix-vector multiply. This is (part of) why the quantization-based approaches are so popular.
- twoodfin 2y ago
  It's memory bound and single threaded?
  - rini17 2y ago
    I found it memory bound so that it was fastest with 6 threads on my 8 thread xeon.
    - twoodfin 2y ago
      Right: So this trick probably doesn’t help much. You only need prefetch when there isn’t enough fetch otherwise.