5 ms·
Run 70B LLM Inference on a Single 4GB GPU with This New Technique
- gardenfelder 3y ago"Large language models require huge amounts of GPU memory. Is it possible to run inference on a single GPU? If so, what is the minimum GPU memory required?"
- deleted 3y ago[deleted]
- sva_ 3y agoNote how they carefully avoid naming an actual tokens/second number for any gpu.
- deleted 3y ago[deleted]
- bronco21016 3y agoAre there any resources that do provide this? I’ve tried on numerous occasions to search for stats on model XYZ puts out x tokens/sec on this hardware setup. It leads me to believe the only way to experiment with running open models is with cloud computing and expensive GPUs and consumer hardware is out. Even for 7B parameters.
- eyegor 3y ago7b can run fine on consumer hardware if you quantize to 4bit. I get around 50-100 token/s on 7b q4 models on older 8gb gpus I have lying around, which is plenty for chat. I don't know of any good resource for "consumer hardware leaderboard" though. In general: 1.3b is about the max you can get for 4gb vram 7b will work on 8gb vram if you quantize 13b will work on 16gb vram if you quantize 33b can work on 24gb if you quanitze If you lack vram but have ram and infinite patience you can do offloading shenanigans with cpu but it tanks performance to unusable levels in my experience.
- instagib 3y agoConsidering quantizing takes a while, care to list some settings guide or a link for people to setup? I use Phind_34B for a 24GB gpu. When in use I typically set gpu memory limit to 22.5GB. Quantization: 1hr 3min. 4bit, group_size=32, desc=true, damp=0.01, trust_remote = false, dtype torch.float16, use_triton = false.
- eyegor 3y agoI'd defer to the llama.cpp or exllamav2 repos for quantizing help, I just use their guides myself. For practical use, I'd say searching for the quantized model on huggingface.co is the way to go. Chances are high that someone else already did what you're interested in and it's a lot faster to download a pre quantized model. Thebloke, turboderp, lonestriker, bartowski. Between these 4 users on huggingface you can find pretty much every common model already quantized.
- avereveard 3y agoWhat do you use to run them? My 3060 is generating just 40token per second with llama.cpp
- eyegor 3y agoI use either llama.cpp or exllamav2 as the backend, but speed is mostly dependent on how many layers fit in your gpu. I'm not sure which gpu I was thinking of for that 50-100 tok/s, on my 2070 I can only get around 15 tok/s on 7b models. I have to use 1.3b models to get to 100 tok/s on this gpu.
- andy99 3y agoSee https://github.com/FMInference/FlexGen https://github.com/FMInference/FlexGen This is from much earlier this year and seems to be the same idea, only loading part of the model into the GPU at a time.
- svnt 3y ago“This new technique” is they load each layer into memory, process the result of that layer, and then clear it and load the next layer. For the cases where this even works, as one commenter said: “I try it on a GTX 1060 6Gb on windows, but I don’t think it’s supposed to be that slow? It took me 13 hours to generate one sentence, RAM was around 9.5Gb and only 6% of GPU in use. It works but I don’t know what I’m doing wrong”
- james-revisoai 3y agoI appreciate your and the other commentators scepticism, and agree for the premise of generating text. However to play devils advocate - Can't inference of single layers be quite useful sometimes for embeddings? and the larger models could embed capabilities only achievable at large parameter size? (i.e. you don't need to finetune the last layer like the BERT days or necessarily use a certain layer for relevant embeddings, you could use the first layer outputs and finetune a custom matrix projection to make some useful embeddings that might include properties of the input that the larger models show capabilities for that smaller models don't?) (ofc this assumes that those capabilities considered "emergent" at larger sizes do not require later layers which they probably do). There is a weird situation where we have the sentence transformer, the bge, e5 and etc kind of embeddings, and then a big jump up to generative model embeddings the model providers provide, but not much widespread adoption of e.g GPT-J or neo-20B embeddings (even though at time of release, they had notable usecases over sentence transformers)
- sp332 3y agoYou can save some space, but it still takes more than one layer to do an embedding.
- svnt 3y agoThere must be some real use for it for sure. Maybe as a result of this effort someone else has their attention called to it and picks up the idea loosely and runs with it in an unexpected way. One of the benefits of lack of contextual or historical understanding is the lack of barriers to action. One of the downsides is you’re likely to do a lot of things of questionable value.
- ramshanker 3y agoI read the whole article but did not find the words / token per second number?
- throwuxiytayq 3y ago~0
- DrSiemer 3y agolower end GPUs like T4 Alright, nothing to see here. I could really use some kind of filter for stuff that does work with a beefy consumer GPU and stuff that does not.
- mattnewton 3y agoHmm? A T4 is less powerful than a “beefy consumer gpu” like a 3080.
- sp332 3y agoWe have tested this code on a 16GB Nvidia T4 GPU. The entire inference process uses less than 4GB GPU memory. You don't need 16GB of VRAM for this, it's just what they had available for testing.
- bhickey 3y ago> Today’s large language models all adopt the Multi-head self-attention structure proposed in Google’s paper “Attention is all you need”. This is what people later call the Transformer structure. Uhh... From the abstract: > We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. —Vaswani, et al. 2017
- peter_d_sherman 3y agoAn excellent article! >"During inference, layers are executed sequentially. The output of the previous layer is the input to the next. Only one layer executes at a time. Therefore, it is completely unnecessary to keep all layers in GPU memory. We can load whichever layer is needed from disk..." ...Or from the RAM of the main computer the Graphics Card is attached to, via the PCIe Bus and DMA (or whatever other methods exist now or in the future)... >"...when executing that layer, do all the calculations, and then completely free the memory after." ...Or keep it in the adjacent RAM of the main computer, for more speed... (Random idea here for hardware designers incidentally: 1) Graphics Card/GPU manufacturers -- put a high-speed CXL (Compute Express Link) interface on your Graphics card (https://en.wikipedia.org/wiki/Compute_Express_Link https://en.wikipedia.org/wiki/Compute_Express_Link). 2) Independent Hardware Designers -- design a stand-alone RAM board (or RAM board rack/tower) which has a high-speed CXL interface on it, which can be interfaced to #1, above. If #1 and #2 exist, now we have Graphics Cards/GPU's that can be interfaced to very large high-speed banks of external RAM -- effectively making the limited local memory of a GPU, a thing of the past... Yes, there might be a slight latency, slightly above that of local GPU memory when accessing this adjacent memory. But I'm guessing that if this latency exists, then over time this will be engineered away... In the short term, well designed software and/or caching strategies could compensate for any latency overhead, if there is one... In conclusion, Graphics Card / GPU + CXL + External Rack o' RAM -- could be a very winning combination...) >"This way, the GPU memory required per layer is only about the parameter size of one transformer layer, 1/80 of the full model, around 1.6GB." A brilliant observation and a brilliantly written article! Well done!
- Arson9416 3y ago[flagged]
- darknoon 3y agoPlease reword this clickbait headline. It's just disk offloading.
- Havoc 3y agoWouldn't that be about as slow as CPU inference?
- angarg12 3y agoProbably much slower, due to all the memory thrashing and lack of pipelining optimizations.
- zaps 3y agoOne Weird Trick, AI Edition
- codedokode 3y agoPlease explain, why GPUs cannot take weights directly from main memory? Why keep weights in VRAM? Why cannot I buy 256 Mb GPU and use it with main RAM?
- dragontamer 3y agoBecause GPUs only have VRAM on board. For GPUs to take data from CPU RAM, it must traverse PCIe which is on the order of ~100x slower than through the on board VRAM.
- haltist 3y agoIt's called direct memory access. The CPU doesn't have to do all I/O. Any subsystem can access RAM with DMA. The main issue is scheduling the loads, stores, and arithemtic operations to avoid stalling the pipeline.
- jamesfmilne 3y agoYes, NVIDIA GPUs can indeed read directly from main memory. It's called Heterogeneous Memory Management. https://developer.nvidia.com/blog/simplifying-gpu-application-development-with-heterogeneous-memory-management/ https://developer.nvidia.com/blog/simplifying-gpu-applicatio... This would allow you to mmap() the weights file into CPU memory, and pass that CPU pointer to the GPU and allow it to fault the data in on-demand. The GPU will have less bandwidth accessing this memory than if you copied it onto the GPU though, as it will need to take a trip over PCIe to read it. But it would eliminate the need to manually upload data to the GPU and the synchronisation involved with that.
- wongarsu 3y agoBecause VRAM is faster than main memory, and a lot faster than ferrying data over PCIe. Nvidia supports reading data directly from main memory. They introduced it around 2013 as Unified Memory, and it occasionally gets updates that make it more usable. It's probably faster than loading each layer into VRAM, computing it, discarding the weights and loading the next layer; but it's still a lot slower than having the weights in VRAM and often slower than just doing the work on the CPU.
- 3y ago
- haltist 3y agoThis is very clever. Now I'm wondering if someone has looked into optimal partitions and instruction streams. This would allow, in theory, to run any AI graph locally by partitioning the graph and loading it one piece at a time. But anyway, if someone has $80B I can get this done. It's just an optimization problem.
- blagie 3y agoI don't like the cynicism here. * Yes, performance numbers would be helpful. * No, this is not new, and for most applications, not practical. The basic point is that yesterday, I couldn't run a large model locally. Today, I can, in about one pip install and 20 lines of code. Yay! There are a lot of things which are convenient to be able to test locally. YES it takes overnight, but if you're trying to confirm if there's a business case, that's often good enough. For a single compute (or even just a few), this is also simpler than spinning up a cloud machine.
- brucethemoose2 3y agoYou can essentially already do this with llama.cpp and ssd swap, much more quickly. Technically it only does the prompt processing and a few layers on the GPU (if that), but honestly that is better, just to avoid all the transfers over the GPU bus. And you can do it in MLC, in your IGP, if you have enough CPU RAM to fit the model. Running a 70B very slowly is nothing new. To be blunt, this strategy is a bad idea in the face of newer implementations.
- cwillu 3y agoNvidea hates him: learn this one weird trick…
- leeeeeepw 3y ago[dead]
- rthnbgrredf 3y agoThis is interesting. While I think this approach may not be practical for a single computer due to poor performance, it could open new possibilities for distributed computing. Here, users across the internet could pool their graphics cards and the 80 layers could be distributed across 80 GPUs, with each GPU handling one layer. The results would then be processed through the system. Admittedly, if there is an average latency of 50 ms for each of the 80 GPUs, it would result in an additional 4 seconds of network delay. However, the overall process could be up to 80 times faster than on a single GPU since the layers are already in the GPU memory, albeit on different cards. This method also has the potential to enable the execution of much larger models on consumer hardware.
- onlywicked 3y agoThere is already an implementation along the same line using the torrent architecture. https://petals.dev/ https://petals.dev/
- alchemist1e9 3y agoDoes any one know the amount of data that is passed between layers? specifically the data size as this information determines the viability of distributed inference.
- jacquesm 3y agoCan't wait for someone who will burn out an SSD drive by memory mapping it and running an LLM on it.
- lostmsu 3y agoSSDs are only weared by writing AFAIK.
- PeterStuer 3y agoI'd say it is not the 4GB GPU setup that is the champion of sharding, but the 4xRTX3090 that is significantly cheaper than a single 80GB vram card. Btw, has anyone experimented with Microsoft's DirectStorage api for this usecase? I understand that the shards will not be very compressable, but maybe the increased IO offloading can have benefits?
- rrhjm53270 3y agoI did roughly the same thing in one of my hobby project https://github.com/fengwang/nnl https://github.com/fengwang/nnl. But in stead of using SSD, I load all the weights to the host memory, and while inferencing the model layer by layer, I asynchronously copy memory from global to shared memory in the hope of better performance. However, my approach is bounded by the PCI-E bandwidth.