3 ms·

So is it running on the phone or not?

• westurner 2y ago
  From the abstract https://arxiv.org/abs/2406.06282 https://arxiv.org/abs/2406.06282 : > Notably, PowerInfer-2 is the first system to serve the TurboSparse-Mixtral-47B model with a generation rate of 11.68 tokens per second on a smartphone. For models that fit entirely within the memory, PowerInfer-2 can achieve approximately a 40% reduction in memory usage while maintaining inference speeds comparable to llama.cpp and MLC-LLM What does this do with a 40 TOPS+ NPU/TPU?