3 ms·

Can this optimization be applied to matmult for us, critters who are running llama on cpu? XD

• twoodfin 2y ago
  I don't think this really helps: It's a trick to drive maximum possible memory bandwidth in service of a single executing thread which can process data as fast as it's being delivered. A parallel matrix multiply running across every core shouldn't have any trouble maximizing memory bandwidth utilization.
  • owlbite 2y ago
    KV-cache based LLM inference is normally significantly memory bound on the matrix-vector multiply. This is (part of) why the quantization-based approaches are so popular.
    • twoodfin 2y ago
      It's memory bound and single threaded?
      • rini17 2y ago
        I found it memory bound so that it was fastest with 6 threads on my 8 thread xeon.
        • twoodfin 2y ago
          Right: So this trick probably doesn’t help much. You only need prefetch when there isn’t enough fetch otherwise.