From their GitHub[0]: > Evaluation shows that PowerInfer attains an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across vari

From their GitHub[0]: > Evaluation shows that PowerInfer attains an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tier server-grade A100 GPU. This significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. Impressive result and very exciting if it holds true! The hybridisation idea (preloading known-"hot" neurons to the GPU and leaving "cold" ones on the CPU) is a neat one on the surface. [0] https://github.com/SJTU-IPADS/PowerInfer https://github.com/SJTU-IPADS/PowerInfer