4 ms·

PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU [pdf]

84 points by georgehill 3y ago

LoganDark 3y ago
> PowerInfer’s source code is publicly available at https://github.com/SJTU-IPADS/PowerInfer https://github.com/SJTU-IPADS/PowerInfer --- Just curious - PowerInfer seems to market itself by running very large models (40B, 70B) on something like a 4090. If I have, say, a 3060 12GB, and I want to run something like a 7B or 13B, can I expect the same speedup of around 10x? Or does this only help that much for models that wouldn't already fit in VRAM?
- voxl 3y ago
  Probably, the idea that the GPU handles the "hot neurons" means that if your GPU is strong enough to handle _enough_ hot neurons, you should expect speedups.
- tofof 3y ago
  The linked paper also includes performance of a second PC configuration. The 'high' configuration is a 4090 (24GB) with a 24-core cpu (iirc), 192 GB ram, pci 4. The 'low' configuration is a 2080ti (11GB) with an 8-core cpu, 64 GB ram, pci 3. The low configuration still averaged a 5x speedup over llama.cpp, so yeah, this is exciting. They noted that for particularly large models (60B+) the hot neurons don't all fit into the gpu's vram and performance falls off. Similarly, for small models with small context sizes enough fits into the gpu to begin with that the performance gains are less pronounced. So there's going to be a sweet spot with regard to the combination of model and context size for a particular configuration, but yes, this still gives a huge speedup compared to llama.cpp. Image of the PC-low configuration's results: https://i.imgur.com/X5reGkd.png https://i.imgur.com/X5reGkd.png showing speedups ranging from 2x-13x.
  - LoganDark 3y ago
    > 24-core cpu Pretty sure it's an 8-core CPU, or at least that's how they're using it. Their demo videos show only 8 threads in use, probably because the "efficiency cores" would negate some of the performance win.
Animats 3y ago
That's an impressive result. LLMs should soon be much cheaper to run.
girvo 3y ago
From their GitHub[0]: > Evaluation shows that PowerInfer attains an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tier server-grade A100 GPU. This significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. Impressive result and very exciting if it holds true! The hybridisation idea (preloading known-"hot" neurons to the GPU and leaving "cold" ones on the CPU) is a neat one on the surface. [0] https://github.com/SJTU-IPADS/PowerInfer https://github.com/SJTU-IPADS/PowerInfer
- 3abiton 3y ago
  They mention OPT-175B, but the currently only available models are Falcon-40B and Llama2 (up to 70B). Curious about the performance in human chatbot arena LLM leaderboard.
Havoc 3y ago
Really clever trick. GPU/CPU splits are currently painfully slow so this may just make them more bearable.
chsasank 3y ago
This is basically fork of llama.cpp. I created a PR to see the diff and added my comments on it: https://github.com/ggerganov/llama.cpp/pull/4543 https://github.com/ggerganov/llama.cpp/pull/4543 One thing that caught my interest is this line from their readme: > PowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed on the CPU, thus significantly reducing GPU memory demands and CPU-GPU data transfers. Apple's Metal/M3 is perfect for this because CPU and GPU share memory. No need to do any data transfers. Checkout mlx from apple: https://github.com/ml-explore/mlx https://github.com/ml-explore/mlx

84 points by georgehill 3y ago

> PowerInfer’s source code is publicly available at https://github.com/SJTU-IPADS/PowerInfer https://github.com/SJTU-IPADS/PowerInfer --- Just curious - PowerInfer seems to market itself by running very large models (40B, 70B) on something like a 4090. If I have, say, a 3060 12GB, and I want to run something like a 7B or 13B, can I expect the same speedup of around 10x? Or does this only help that much for models that wouldn't already fit in VRAM?

voxl 3y ago
Probably, the idea that the GPU handles the "hot neurons" means that if your GPU is strong enough to handle _enough_ hot neurons, you should expect speedups.
tofof 3y ago
The linked paper also includes performance of a second PC configuration. The 'high' configuration is a 4090 (24GB) with a 24-core cpu (iirc), 192 GB ram, pci 4. The 'low' configuration is a 2080ti (11GB) with an 8-core cpu, 64 GB ram, pci 3. The low configuration still averaged a 5x speedup over llama.cpp, so yeah, this is exciting. They noted that for particularly large models (60B+) the hot neurons don't all fit into the gpu's vram and performance falls off. Similarly, for small models with small context sizes enough fits into the gpu to begin with that the performance gains are less pronounced. So there's going to be a sweet spot with regard to the combination of model and context size for a particular configuration, but yes, this still gives a huge speedup compared to llama.cpp. Image of the PC-low configuration's results: https://i.imgur.com/X5reGkd.png https://i.imgur.com/X5reGkd.png showing speedups ranging from 2x-13x.
- LoganDark 3y ago
  > 24-core cpu Pretty sure it's an 8-core CPU, or at least that's how they're using it. Their demo videos show only 8 threads in use, probably because the "efficiency cores" would negate some of the performance win.

Animats 3y ago

That's an impressive result. LLMs should soon be much cheaper to run.

girvo 3y ago

From their GitHub[0]: > Evaluation shows that PowerInfer attains an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tier server-grade A100 GPU. This significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. Impressive result and very exciting if it holds true! The hybridisation idea (preloading known-"hot" neurons to the GPU and leaving "cold" ones on the CPU) is a neat one on the surface. [0] https://github.com/SJTU-IPADS/PowerInfer https://github.com/SJTU-IPADS/PowerInfer

3abiton 3y ago
They mention OPT-175B, but the currently only available models are Falcon-40B and Llama2 (up to 70B). Curious about the performance in human chatbot arena LLM leaderboard.

Havoc 3y ago

Really clever trick. GPU/CPU splits are currently painfully slow so this may just make them more bearable.

chsasank 3y ago

This is basically fork of llama.cpp. I created a PR to see the diff and added my comments on it: https://github.com/ggerganov/llama.cpp/pull/4543 https://github.com/ggerganov/llama.cpp/pull/4543 One thing that caught my interest is this line from their readme: > PowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed on the CPU, thus significantly reducing GPU memory demands and CPU-GPU data transfers. Apple's Metal/M3 is perfect for this because CPU and GPU share memory. No need to do any data transfers. Checkout mlx from apple: https://github.com/ml-explore/mlx https://github.com/ml-explore/mlx