3 ms·
A few months back, the PyTorch team released GPT, Fast (https://news.ycombinator.com/item?id=38477197 https://news.ycombinator.com/item?id=38477197), a collecti
by MDK8888 3y ago
A few months back, the PyTorch team released GPT, Fast (https://news.ycombinator.com/item?id=38477197 https://news.ycombinator.com/item?id=38477197), a collection of techniques to 10x the inference speed of Llama-2-7b.
This library generalizes those techniques (quantization, torch.compile, speculative decoding) to all Huggingface models, achieving an inference speed speedup of 6-7x.
As more optimization algorithms are discovered, I will post updates to the repo to HN.
- theboat 3y agodoes this work for diffusion models, e.g. stable diffusion xl?
- jurgenaut23 3y agoA quick look at the codebase seems to indicate that it doesn't. This seems very targeted at LLMs.
- mkesper 3y agoPlease add (something like) that to the Readme.
- ipsum2 3y agoPretty confusing that both the pytorch repo and yours is named gpt fast.