According to this [1] article (current top of hn) memory bandwidth is typically the limiting factor, so as long as your batch size isn't huge you probably aren'

According to this [1] article (current top of hn) memory bandwidth is typically the limiting factor, so as long as your batch size isn't huge you probably aren't losing too much in performance. https://finbarr.ca/how-is-llama-cpp-possible/ https://finbarr.ca/how-is-llama-cpp-possible/