9 ms·
Accelerating Generative AI with PyTorch II: GPT, Fast
- brucethemoose2 3y agoThis is similar to exllamav2, and exllamav2's quantization is also excellent.
- AmazingTurtle 3y ago240tok/s is crazy
- chillee 3y agoHey, author of the blog post here. It's mentioned in the blog post, but one of the intentions of this repo is that it's more of a "tutorial" than it is a library/framework. My hope is that people will copy-paste and modify it for their own needs :) Code can also be found here: https://github.com/pytorch-labs/gpt-fast https://github.com/pytorch-labs/gpt-fast And a twitter thread summary here: https://twitter.com/cHHillee/status/1730293330213531844 https://twitter.com/cHHillee/status/1730293330213531844
- buildbot 3y agoGreat work and a really useful resource! Comprehensive guides on improving PyTorch performance are pretty hard to come by, and I learned a couple new tricks from this!
- ilaksh 3y agoWhat GPU was used when testing this? Is this faster than HuggingFace's Text Generation inference container?
- chillee 3y agoWe used an A100-80GB GPU. We didn't compare explicitly to Huggingface TGI but I think you should be able to compare the tokens/s achieved. One note is that this release is optimized for latency, while I think HF TGI might be more optimized for throughput.
- smith7018 3y agoGreat work! Do you know if it's possible to port this over to pytorch's Apple Silicon/MPS support?
- chillee 3y agoUnfortunately it's a little bit tricky today. The main issue is that we rely heavily on torch.compile + Triton for performance in this repo, and there isn't an Apple Silicon backend either for torch.compile or Triton. For example, there's an AMD backend for Triton (and it's also integrated into torch.compile), which is why we can mostly do the same optimizations on Nvidia and AMD GPUs. Ideally, there'd be an Apple Silicon backend for Triton, and then this repo would mostly work out of the box :)
- Dowwie 3y agoWhat kind of workstation would you build/buy for local GPT development with a budget of $3000? Is remote dev a viable alternative to local workstations?
- woodson 3y agoI’d go with a remote dev solution. Training/finetuning of large models requires much more resources anyway, so the GPUs in the local machine would be unused most of the time.
- leobg 3y agoNot OP, but I asked myself that same question two years ago. Then I looked at the energy prices in Germany and knew I had no chance against cloud GPUs. Maybe you live in a country with lower energy prices, like Bermuda (or any other country on earth), in which case this may not be as important to you. A side benefit of going cloud that you can pick and choose the right GPU for whatever project you’re working on, and you’re really just paying while you’re running them. Also, no hardware or Cuda drivers that may divert your attention.
- ftufek 3y agoLocal workstation is much cheaper in the long run. Even ignoring that, most of the development is running experiments. You're gonna be hesitant to run lots of experiments if they each cost money whereas when you pay upfront for the hardware, you're gonna have the incentive to fully utilize it with lots of experiments. I'd go with rtx 4090 and deal with memory limitation through software tricks. It's an underrated card that's as performant as cards that are magnitude pricier. It's great way to get started with that budget.
- Philpax 3y agoDepending on what you're doing, 2x used 3090s are the same price and offer you more VRAM. That's what I'm planning on doing, in any case - being able to run 70B LLMs entirely on the GPU is more useful than being able to run 34B faster.
- wolftickets 3y agoJust wanted to share, the charts and gifs are exceptionally well done. Informative, concise, and easy to read.
- chillee 3y agoThanks! I've also written a couple other things along a similar vein you might like at https://horace.io/writing.html https://horace.io/writing.html (particularly https://horace.io/brrr_intro.html https://horace.io/brrr_intro.html) and also some of the things I've tweeted: https://twitter.com/cHHillee/highlights https://twitter.com/cHHillee/highlights
- toxik 3y agoFantastic article, great job. One small note: eke and eek are not the same word.
- _giorgio_ 3y agoWhat's the difference between, say, Karpathy's nanoGPT and your GPT implementation? Is it just a difference in speed, or are there some new theory details to learn? Thanks for sharing your work.
- chillee 3y agoJust a difference in speed. This repo is primarily showing how you can get really good inference perf with just native pytorch.
- kartoolOz 3y agoHi, Thanks for Open sourcing the code! I was trying to reuse the code especially the dynamic quantization per channel (int8 on gpu) but couldn't get it to work, i also checked out torchao package but it looks like it has dependency on the nightly channel and SAM's dynamic implementation with triton has other issues, is there any clean implementation of int8 dynamic post-training quantization that you can point too ?
- chillee 3y agoWhat’s the issue with getting int8 dynamic quantization to work? As in, you’re unable to get it to quantize or to run with speedups?
- xmichael909 3y agoHoly hotdogs, this look amazing. So ahh. I'll jump right to it - where can I run this online without having to do a bunch of work setting it up? I have several python projects that could take advantage of this! (;
- andy99 3y agoThis is a great article. Regarding > While these projects are performant, they often come with tradeoffs in ease of use, such as requiring model conversion to specific formats or building and shipping new dependencies. I think it should be acknowledged that (at least IMO) pytorch model formats are not very portable and this is a big part of the problem. It would be nice to see industry move towards a better format (gguf?) that can easily be ported between frameworks and not leave you stuck using torch to load it. Likewise, pytorch is a massive dependency to include with a project, especially for simple inference, so while other projects have new dependencies, they can often be a lot lighter than for a pytorch model, again particularly for inference code.
- chillee 3y agoYeah, for sure. I think for deployment purposes, many times these model conversions are necessary (such as if you don't want to use Python). However, I do think these model conversions are often a significant pain for users. So, in some sense, the goal here is to show that the performance component and the "convert your model for deployment" component can be disentangled. We also have work on allowing you to "export" an AOT-compiled version of your model with torch.compile, and that should allow you to deploy your models to run in other settings.
- andy99 3y agoThanks for the reply. "show that the performance component and the "convert your model for deployment" component can be disentangled" makes sense. Also, I liked the part of the article about torch.compile producing faster matrix-vector multiplication than cublas. I've seen the same thing on CPU, that it's way faster to just write and manually optimize a loop over a bunch of dot products than it is to use BLAS routines because of how simple the "matmul" actually is. I don't know how widely known that is.
- dnnssl2 3y agoWhat are some of the better use cases of fast inference? From my experience using ChatGPT, I don't need it to generate faster than I can read, but waiting for code generation is painful because I'm waiting for the whole code block to format correctly, be available to copy or execute (in the case of code interpreter). Anything else fall under this pattern?
- wedn3sday 3y agoOne obvious use case is that it makes per-token generation much cheaper.
- dnnssl2 3y agoThat's not so much a use case, but I get what you're saying. It's nice that you can find optimizations to shift down the pareto frontier of across the cost and latency dimension. The hard tradeoffs are for cases like inference batching where it's cheaper and higher throughput but slower for the end consumer. What's a good use case for an order of magnitude decrease in price per token? Web scale "analysis" or cleaning of unstructured data?
- jasonjmcghee 3y agoProgrammatic and multi-step use cases. If you need chain-of-thought or similar, tool use, etc. Generating data. Most use cases outside of classic chat. For example, I made an on-demand educational video project, and the slowest part was by far the content generation. RAG, TTS, Image generation, text rendering, and video processing were all a drop in the bucket, in comparison. It would be an even wider gap now, and TTS is super-realtime, and image generation can be single step.
- rfw300 3y agoThe main thing is chat is just one application of LLMs. Other applications are much more latency sensitive. Imagine, for instance, an LLM-powered realtime grammar checker in an editor.
- ClarityJones 3y agoPerhaps this is naive, but in my mind it can be useful for learning. - Hook LLM to VMs - Ask for code that [counts to 10] - Run code on VM - Ask different LLM to Evaluate Results. - Repeat for sufficient volume. - Train. The faster it can generate results the faster those results can be tested against the real world, e.g. a VM, users on X, other models with known accuracies.
- dnnssl2 3y agoIf you were to serve this on a datacenter server, is the client to server roundtrip networking the slowest part of the inference? Curious if it would be faster to run this cloud GPUs on better hardware but farther compute, or locally with worse hardware.
- chillee 3y agoSurprisingly, no. And part of this is that text generation is really expensive. Unlike traditional ML inference (like with, resnets), you don't just pass your data through your model once. You need to pass it over and over again (once for each token you generate). So, in practice, a full "text completion request" can often take on the order of seconds, which dwarfs the client <-> server roundtrip.
- dnnssl2 3y agoIs this still the case for sliding window attention/streaming LLMs, where you have a fixed length attention window rather than infinitely passing in new tokens for quadratic scaling? You even get better performance due to purposely downsampling non-meaningful attention sink tokens.
- chillee 3y agoI cover it a bit in the blog post, but unless you have a really long context length (like 32k+), your primary computational cost doesn't come from attention but rather from loading your weights from VRAM into registers. I mean, practically speaking, completions from say, ChatGPT or Claude take seconds to finish :)
- dnnssl2 3y agoHow does one select a good candidate for the draft model in speculative decoding? I imagine that there's some better intuition than just selecting the next parameter count down (i.e 70B -> 13B, 13B -> 7B). Also how does that interact with MoE models? Do you have a mini version of the MoE, with smaller experts?
- chillee 3y agoThis is indeed a bit of a dark art. Essentially, you want a balance between "is significantly faster than base model" and "generates similar stuff to the base model". Anecdotally, folks often seem to use say, 70B base + 7B as verifier. But I think there's a lot of room for experimentation and improvement here. You could... say, take a 70B model and maybe just chop off the last 90% of layers and then fine-tune. Or perhaps you could use a model that's trained to generate 8 tokens at once. Or perhaps you could just use statistical "n-gram" predictor.
- claytonjy 3y agoOne of the notable tricks the various LLM serving frameworks provide is a special approaches to batching, e g. continuous, persistent, or in-flight batching depending on the inference framework. At some level they each allow you to start a new generation while in the middle of one or more previous generations. Is that possible with "just" pytorch? Could it be added to gpt-fast?
- chillee 3y agoYeah it's certainly possible, but it's not the focus of this implementation, which is more latency focused (so BS=1).
- lmeyerov 3y agoyeah i'm curious how this would stack up to vllm in a batch setting long-term, bodes well as both methodologies should be combineable, just curious for current point-in-time wrt being relevant for production use
- chillee 3y agoI wouldn't recommend using it for a batch serving setting today. One crucial optimization for batched serving (which you need if you have a large number of requests) is continual batching, which this implementation doesn't have.
- sva_ 3y agoOfftopic, but what software do they use to create the benchmark flamegraphs? I've been using cProfile with snakeviz, but am curious to try alternatives.
- chillee 3y agoAssuming you're talking about the one here: https://pytorch.org/blog/accelerating-generative-ai-2/#starting-point-255-toks https://pytorch.org/blog/accelerating-generative-ai-2/#start... it's just the pytorch profiler + chrome profiler (chrome://tracing)
- glandium 3y agoThe screenshot in the post looks like https://www.chromium.org/developers/how-tos/trace-event-profiling-tool/ https://www.chromium.org/developers/how-tos/trace-event-prof...
- syrusakbary 3y agoI'd love to see how this compares against Llama.cpp on speed. Do anyone any benchmarks that compare PyTorch.compile vs Llama.cpp?
- chillee 3y agoThis person claims to have compared llama.cpp against gpt-fast on a 4090, and found gpt-fast about 20% faster. https://twitter.com/zeuxcg/status/1730450360895242355 https://twitter.com/zeuxcg/status/1730450360895242355
- syrusakbary 3y agoAwesome, thanks!
- deepGem 3y ago"This may sound implausible to many of you, considering how hard it is to write efficient matrix multiplication/attention kernels, and how much manpower has been put into CuBLAS and FlashAttention. The key here, however, is that transformer decoding has very unusual computational properties. In particular, because of the KV-cache, for BS=1 every single matrix multiplication in a transformer is actually a matrix vector multiplication." How did they unlock this key ? In retrospect it seems so simple, but without the KV-cache this possibility would not have emerged at all. Hats off !
- cztomsik 3y agoBS=1 also means you can only decode one token. This is not always what you want, and then you're back to matrix-matrix multiply again. Most of the from-scratch llama implementations start with matrix-vector because it's simpler & easier. But eventually, you will want to serve multiple users, or compute multiple beams together...
- kosolam 3y agoHey we are exactly dealing now with all that is related to the performance of running models from HF. Yesterday we ran vicuna-13b-q8-gguf using llamacpp on a vast.ai A40 45GB VRAM It gave us 4 tokens/s generation rate. This seems a bit slow for that GPU and a 13b model. Does anyone know where the problem could be in llamacpp, gpu, the model, something else? Also… where are all the people like us that work on applications on top of HF models congregate?
- bicepjai 3y agoCan we get same performance on Apple gpu ?
- two_in_one 3y agoGreat work. Actually it's sort of breakthrough because it makes interesting things possible. (if it's not too late...)
- maxloo1976 3y agoI'm wondering if gpt-fast has a version that can be run from Windows Command Prompt or Powershell? https://github.com/pytorch-labs/gpt-fast/issues/45 https://github.com/pytorch-labs/gpt-fast/issues/45