3 ms·

I suggest figuring out what your configuration problem is. Which llama.cpp flags are you using, because I am absolutely not having the same bug you are.

• EnPissant 1y ago
  It's not a bug. It's the reality of token generation. It's bottlenecked by memory bandwidth. Please publish your own benchmarks proving me wrong.
  • DiabloD3 1y ago
    I cannot reproduce your bug on AMD. I'm going to have to conclude this is a vendor issue.