2 ms·
From the abstract https://arxiv.org/abs/2406.06282 https://arxiv.org/abs/2406.06282 : > Notably, PowerInfer-2 is the first system to serve the TurboSparse-Mixt
by westurner 2y ago
From the abstract https://arxiv.org/abs/2406.06282 https://arxiv.org/abs/2406.06282 :
> Notably, PowerInfer-2 is the first system to serve the TurboSparse-Mixtral-47B model with a generation rate of 11.68 tokens per second on a smartphone. For models that fit entirely within the memory, PowerInfer-2 can achieve approximately a 40% reduction in memory usage while maintaining inference speeds comparable to llama.cpp and MLC-LLM
What does this do with a 40 TOPS+ NPU/TPU?