4 ms·
Just a note that you have to have at least 12GB VRAM for it to be worth even trying to use your GPU for LLaMA 2. The 7B model quantized to 4 bits can fit in 8G
by lolinder 3y ago
Just a note that you have to have at least 12GB VRAM for it to be worth even trying to use your GPU for LLaMA 2.
The 7B model quantized to 4 bits can fit in 8GB VRAM with room for the context, but is pretty useless for getting good results in my experience. 13B is better but still not anything near as good as the 70B, which would require >35GB VRAM to use at 4 bit quantization.
My solution for playing with this was just to upgrade my PC's RAM to 64GB. It's slower than the GPU, but it was way cheaper and I can run the 70B model easily.
- dc443 3y agoI have 2x 3090 do you know if it's feasible to use that 48GB total for running this?
- eurekin 3y agoYes, it runs totally fine. I ran it in Oobabooga/text generation web ui. Nice thing about it is that it autodownloads all necessary gpu binaries on it's own and creates a isolated conda env. I asked same questions on the official 70b demo and got same answers. I even got better answers with ooba, since the demo cuts text early Ooobabooga: https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui Model: TheBloke_Llama-2-70B-chat-GPTQ from https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ ExLlama_HF loader gpu split 20,22, context size 2048 on the Chat Settings tab, choose Instruction template tab and pick Llama-v2 from the instruction template dropdown Demo: https://huggingface.co/blog/llama2#demo https://huggingface.co/blog/llama2#demo
- zakki 3y agoIs there any specific settings to make 2x3090 work together?
- kwerk 3y agoInterested in this too
- eurekin 3y agoNot really? I just got those cards in separate PCI slots and the Exllama_hf handles spreading the load internally. No NVLink bridge in particular. I use the "20,22" memory split so that the display card has some room for the framebuffer to handle display
- olavfosse 3y agoI'm very curious what your other components are and how you managed to fit 2 3090s in one PC.
- ErneX 3y agoApple Silicon Macs might not have great GPUs but they do have unified memory. I need to try this on mine I have 96GB of RAM on my M2 Max.
- diffeomorphism 3y agoWhat does "unified" actually mean and how much would that help? It is still off the shelf LPDDR5‑6400, just with a better interconnnect (like a ps5). How does this compare to non-unified ddr5 or hmb2e as on nvidia A100 cards?
- thejosh 3y agoIt's a term used to justify non replaceable parts :-).
- lhl 3y agoThe benefits are primarily price - 96GB of VRAM would be 4x3090/4090 (~$6K) or 2xA6000 (~$8-14K) cards (also, looks like you can buy an 80GB A100 PCIe for about $15K atm). While Apple is using LPDDR5, it is also running a lot more channels than comparable PC hardware. The M2 has 100GB/s, M2 Pro 200GB/s, M2 Max 400GB/s, and M2 Ultra is 800GB/s (8 channel) of memory bandwidth. The Nvidia cards are about 900GB/s-1TB/s (A100 PCIe gets up to 1.5TB/s). In practice, on quantizes of the larger open LLMs, an M2 Ultra can currently inference about 2-4X faster than the best PC CPUs I've seen (mega Epyc systems), but also about 2-4X slower than 2x4090s.
- diffeomorphism 3y agoThat is useful info, but still does not quite address the question. The question was how memory type, memory amount and bandwidth factor into actual performance. So let me rephrase: Given a budget of $X, what performance/limitations should you expect with - 256GB of non-unified DDR5 in a PC, just CPU - 128GB of DDR5 for an APU - 96GB of unified DDR5 - Whatever Nvidia will sell you for $X. An answer of "just compare a single memory bandwidth number" seems a bit short. Sure, more bandwidth helps, but is half as much RAM at double bandwidth better or worse?
- NoMoreNicksLeft 3y agoTrying to figure out what hardware to convince my boss to spend on... if we were to get one of the A6000/48gb cards, will that see significant performance improvements over just a 4090/24gb? The primary limitation is vram, is it not?
- lolinder 3y agoVRAM is what gets you up to the larger model sizes, and 24GB isn't enough to load the full 70B even at 4 bits, you need at least 35 and some extra for the context. So it depends a lot on what you want to do—fine tuning will take even more as I understand it. The card's speed will affect your performance, but I don't know enough about different graphics cards to tell you specifics.
- ycombmehair 3y agoHow would an APU, such as 5700g with up to 128gb of system ram perform when allocating it as vram? Is this a cost effective way of using running this on a budget?
- NoMoreNicksLeft 3y agoWell, 48gb is better than nothing at least. And it has the potential (if we get the build right) to drop a second A6000 card into it with the nvlink module (I think this does allow you to effectively have 96gb) later.
- cjbprime 3y agoYou might consider getting a Mac Studio (with as much RAM as you can afford up to 192GB) instead, since 192GB is more (unified) memory than you're going to easily get to with GPUs.
- abhibeckert 3y agoThis. The main system memory on a Mac Studio is GPU memory and there's a lot of it. It also has the Neural Engine, which is specifically designed for this type of work - most software isn't designed to take advantage of that yet, but presumably it will soon.
- flangola7 3y agoWhat is necessary to run 70B on CPU without quantization?
- ewokone 3y agoBump. interested as well.
- esperent 3y agoDoes that mean you have to run it on the CPU? Or can you use the GPU with system RAM?
- creata 3y agoRunning just some of the layers on the GPU can still make things much faster, though.
- lhl 3y agoJust a quick note, it's worth pointing out that for most people (eg, wanting to chat to a model in realtime), I don't think running locally on a CPU is a very viable option unless you're very patient. On my 16c Ryzen 5950X/64GB DDR4-3800 system, llama-2-70b-chat (q4_K_M) running llama.cpp (eb542d3) and testing doing a 100 token test (life's too short to try max context), I got 1.25 tokens/second (~1 word/second) output. Compiled with cuBLAS w/ `-ngl 0` (~400MB of VRAM usage, no layers loaded) makes no perf difference. The max layers I can load on a headless 24GB 4090 is 45/83 layers (running `-ngl 45 --low-vram`) which brings speeds up to 2.5 t/s. A little less painful, but still not super pleasant. For reference, people have reported performance of 12-15 t/s with 2x4090s w/ exllama (GPTQ). People are using a 14,20 split and able to load a full (NTK Rope scaled) 16K context into 48GB of VRAM.