3 ms·

> PowerInfer’s source code is publicly available at https://github.com/SJTU-IPADS/PowerInfer https://github.com/SJTU-IPADS/PowerInfer --- Just curious - Power

> PowerInfer’s source code is publicly available at https://github.com/SJTU-IPADS/PowerInfer https://github.com/SJTU-IPADS/PowerInfer --- Just curious - PowerInfer seems to market itself by running very large models (40B, 70B) on something like a 4090. If I have, say, a 3060 12GB, and I want to run something like a 7B or 13B, can I expect the same speedup of around 10x? Or does this only help that much for models that wouldn't already fit in VRAM?

voxl 3y ago
Probably, the idea that the GPU handles the "hot neurons" means that if your GPU is strong enough to handle _enough_ hot neurons, you should expect speedups.
tofof 3y ago
The linked paper also includes performance of a second PC configuration. The 'high' configuration is a 4090 (24GB) with a 24-core cpu (iirc), 192 GB ram, pci 4. The 'low' configuration is a 2080ti (11GB) with an 8-core cpu, 64 GB ram, pci 3. The low configuration still averaged a 5x speedup over llama.cpp, so yeah, this is exciting. They noted that for particularly large models (60B+) the hot neurons don't all fit into the gpu's vram and performance falls off. Similarly, for small models with small context sizes enough fits into the gpu to begin with that the performance gains are less pronounced. So there's going to be a sweet spot with regard to the combination of model and context size for a particular configuration, but yes, this still gives a huge speedup compared to llama.cpp. Image of the PC-low configuration's results: https://i.imgur.com/X5reGkd.png https://i.imgur.com/X5reGkd.png showing speedups ranging from 2x-13x.
- LoganDark 3y ago
  > 24-core cpu Pretty sure it's an 8-core CPU, or at least that's how they're using it. Their demo videos show only 8 threads in use, probably because the "efficiency cores" would negate some of the performance win.

3 ms·

> PowerInfer’s source code is publicly available at https://github.com/SJTU-IPADS/PowerInfer https://github.com/SJTU-IPADS/PowerInfer --- Just curious - Power

by LoganDark 3y ago

voxl 3y ago
Probably, the idea that the GPU handles the "hot neurons" means that if your GPU is strong enough to handle _enough_ hot neurons, you should expect speedups.
tofof 3y ago
The linked paper also includes performance of a second PC configuration. The 'high' configuration is a 4090 (24GB) with a 24-core cpu (iirc), 192 GB ram, pci 4. The 'low' configuration is a 2080ti (11GB) with an 8-core cpu, 64 GB ram, pci 3. The low configuration still averaged a 5x speedup over llama.cpp, so yeah, this is exciting. They noted that for particularly large models (60B+) the hot neurons don't all fit into the gpu's vram and performance falls off. Similarly, for small models with small context sizes enough fits into the gpu to begin with that the performance gains are less pronounced. So there's going to be a sweet spot with regard to the combination of model and context size for a particular configuration, but yes, this still gives a huge speedup compared to llama.cpp. Image of the PC-low configuration's results: https://i.imgur.com/X5reGkd.png https://i.imgur.com/X5reGkd.png showing speedups ranging from 2x-13x.
- LoganDark 3y ago
  > 24-core cpu Pretty sure it's an 8-core CPU, or at least that's how they're using it. Their demo videos show only 8 threads in use, probably because the "efficiency cores" would negate some of the performance win.