11 ms·
Exllamav2: Inference library for running LLMs locally on consumer-class GPUs
- msp26 3y agoI know it's not true but part of me believes that meta intentionally held back the 34B to make the community work on improving 13B and 70B model usage.
- brucethemoose2 3y agoAmazing. But at what cost? Have there been any perplexity benchmarks for ELX2? Its annoying that Facebook made llama v2 70B instead of 65B, even with the memory saving changes... 5B less parameters, and the squeeze would be far easier.
- Palmik 3y agoGood point -- hopefully the quality impact is still worth it, remains to be seen. Agree on the size -- hopefully something they will keep in mind for future models.
- brucethemoose2 3y agoIf its better than the equivalent 30B model, that's still a huge achievement. Llama.cpp's Q2_K quant is 2.5625 bpw with perplexity just barely better than the next step down: https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 But subjectively, the Q2 quant "feels" worse than its high wikitext perplexity would suggest. That's apples to oranges, as this quantization is different than Q2_K, but I just hope the quality hit in practice isn't so bad.
- Palmik 3y agoWhy this is interesting: To my knowledge, this is the first time you are able to run the largest Llama, at competitive speed, on a consumer GPU (or something like A40). On many tasks, fine-tuned Llama can outperform GPT-3.5-turbo or even GPT-4, but with a naive approach to serving (like HuggingFace + FastAPI), you will have hard time beating the cost of GPT-3.5-turbo even with the smaller Llama models, especially if your utilization is low.
- politelemon 3y agoCould you explain what you mean by naive approach to serving?
- Palmik 3y agoSure, there are many ways one can be accidentally inefficient, some examples: Batching: Your throughput (and thus cost per token) will be much worse if you don't do batching (meaning running inference for several inputs at once). To do batching, you either need to have a high sustained QPS or adjust your workload to be bursty (but this has utilization issues). Inference engine: The regular HuggingFace, even with built in optimizations, is not competitive against inference engines like vllm, exllama, etc. Utilization: Depending on your scale, it might be hard for you to have a nice flat utilization, or to even utilize one GPU at 100% capacity. This means you need to solve scaling up & down your machines and the issues associated with it (can you live with cold starts? etc.) Hardware: Forget server-less GPUs like Replicate etc, their markups compared to on-demand pricing is usually >10x. On-demand A100 or H100, provided you can even get a quota for them, are also expensive. Spot instances are better. Older GPUs (A10, T4, etc.) are better. Your own GPU cluster is likely the best if you have the scale (and likely, you can resell your cluster within the next 18months without much if any depreciation). For these reasons, I have been toying with the idea of providing a dead simple service for fine-tuning & serving open-source LLMs where the users actually own (and can download) the weights. If anyone is interested in this and would like to chat, let me know.
- TheIronYuppie 3y agoWe’d be interested in helping! aronchick (at) expanso (dot) io
- quickthrower2 3y agoInterested in this. Are you thinking of it being a SaaS type thing? I have some rudimentary experience, i.e. training NanoGPT on both Modal.com and Vast.ai (which is vastly cheaper, but privacy/cost trade off there). It could be a layer on top of $gpu_cloud (so end user signs up for the cloud) and the service makes use of those resources.
- pulse7 3y agoHow does "70B Llama 2" compare with ChatGPT 3.5/4.0 in reality? Are they on par? In all areas?
- andai 3y agoIt seems to depend on the task. I'd say it beats GPT-3 in quality most of the time (but not in speed and cost!) and GPT-4 approximately never. It's perfect if what you need is "less good than GPT-4 but 50x cheaper."
- wkat4242 3y agoThe biggest benefit of llama is that you can get it uncensored so you don't get the weasely lawyer disclaimer crap all the time.
- andai 3y agoIn my experience the Chat version of Llama 2 is significantly worse than the GPTs, on the "I'm afraid I can't do that" front. It doesn't lecture you as much, but suggests "how about we do [something unrelated and useless] instead?" As for the plain text prediction version (which is the only uncensored one?), I haven't been able to get it to do anything useful, even when I provide examples (seems dumber than even ancient GPT-3?). Also, I got some bizarre and disturbing outputs from the uncensored version, like it was trained on some very nasty inputs! I assume that's why they went so hard on the safety phase to compensate...
- brucethemoose2 3y agoSome 70B finetunes excel at certain niches, like specific non-English languages, roleplay, fictional writing, or topics GPT4 would refuse to discuss. Some of these are hard to evaluate, but try out (for instance) MythoMax for fiction writing, Airoboros for "uncensored" general use, and Samantha for therapist style chat. https://huggingface.co/models?sort=modified&search=70b https://huggingface.co/models?sort=modified&search=70b
- 3y ago
- CodeCompost 3y agoWhat about the 4070 which 'only' has 12GB of VRAM?
- brucethemoose2 3y agoThats a good fit for 13B at the moment. Or older 30B models with CPU offloading. I'm not certain 30B models will fit completely on 12GB, even with this quantization.
- CodeCompost 3y agoMaybe a stupid question, I'm not familiar with how video ports work, but does the concept of memory swapping exist for video cards? Or is that physically impossible? Obviously that would be useless for games, but for LLMs it may be an option?
- rwmj 3y agoIf you need to use all the memory then swapping breaks down, best you can do is try to divide the work so half can be done, then you completely swap all the memory and do the other half. Also PCIe is (relatively) slow for moving gigabytes of data back and forth.
- rahimnathwani 3y agoSwapping would be too slow. Llama.cpp allows you to specify that n layers load into the GPU's VRAM, and the rest load into main memory.
- brucethemoose2 3y agoSome frameworks have tried to do this automatically, but its extremely slow. You are better off just running whatever won't fit on the CPU (which is what llama.cpp does).
- kilnr 3y agoI get about the same speed (35 tok/s) out of 13B Llama2 on a 4070 Ti, FWIW.
- ftxbro 3y agoThis is quantized down to 2.5 bits per weight, whereas single precision accuracy is 32 bits.
- tutfbhuf 3y agoHow is 2.5 bits possible? Is it an average between 2 and 3 over all weights?
- ftxbro 3y ago> Is it an average between 2 and 3 over all weights? Yes I think it's an average where different quantization levels are used for different layers or weights. Here are more details about the quantization scheme: https://github.com/turboderp/exllamav2#exl2-quantization https://github.com/turboderp/exllamav2#exl2-quantization
- 3abiton 3y agoAny benchmarks on performance?
- redox99 3y agoThese models aren't even trained on FP32. The original format is FP16. And quantizing to INT8/FP8 is almost lossless. But yes, 2.5 bits per weight is pretty insane.
- version_five 3y agoThis should be in the headline (the 2.5 bit part), without the qualifier the result means nothing.
- yieldcrv 3y agoWhat’s a useful amount of tokens/second? What’s a speed like ChatGPT 4? I’m trying to understand if there are agreed upon metrics such as “frames per second” in other fields or niches
- brucethemoose2 3y agoMore than ~10 tokens/s streamed feels OK to read. I can kinda live with 4-5 tokens/s (which is what I get on llama 70B on my 3090 with llama.cpp CPU offload). What feels "good" depends on the person and the type of content, but 35 tokens/s should feel very fast.
- enjeyw 3y agoI wonder if anyone will end up employing the "human" solution of padding sentences out with filler words like 'Ummmm' or 'So like' while the generator is determining the actual next token...
- visarga 3y agoOnce the model finished reading the prompt and starts answering tokens are generated at constant speed, there are no Ummm... gaps. But some tokens are already decided from the first layers and other tokens need the full depth. So there is a notion of "thinking harder" on particular tokens. We could detect token difficulty and fake the output to show the pauses.
- sigmoid10 3y ago>What’s a useful amount of tokens/second? Depends entirely on your application. If you want a real time chatbot, anything more than 10 tokens/s is probably generating text faster than the user can read, so it's fine. Do you want code suggestions or completions? Probabaly also fine, but a bit slow. Do you want to create summaries for 1000 documents? That's gonna be really slow. But at this frontier of the field, tokens/s is not really the issue. Performance vs. quality is. If you lose a whole lot of accuracy by quantizing floats down to single digit precisions but in turn are able to run 70B parameter models, you often still get better results than less thoroughly quantized 7B parameter models. It's all about getting big models to run on memory limited GPUs.
- ppsreejith 3y agoIIUC this should work on the RTX 3090 as well (probably at less than 35tps)? Since the minimum requirement seems to be 24GB of VRAM
- mejutoco 3y agoIn the table linked it says for "V2: 3090Ti" 30 t/s
- jensb1 3y agoCould someone explain how token generation speed relates to latency for the first token to be outputted? And if anyone have any metrics on latency on a 4090 for the 70B model, that would be very helpful.
- MezzoDelCammin 3y ago4090Ti? AFAIK that has never been released
- jensb1 3y agotypo
- redox99 3y agoUnrelated. What matters for that is prompt processing time (which is in the high hundreds of tokens per second).
- MezzoDelCammin 3y agoInteresting bit at the end of the text: 2023-09-13: Preliminary ROCm support added Makes me curious how will the RTX4090/3090 compare with something 7900-ish
- lhl 3y agoWhile I haven't given ExLLamaV2 a try yet since my 7900XT is on a Win10 dedicated gaming machine, I recently sat down and figured out getting llama.cpp w/ ROCm running on it: https://llm-tracker.info/books/howto-guides/page/amd-gpus#bkmrk-compile-for-rocm https://llm-tracker.info/books/howto-guides/page/amd-gpus#bk... Here's how it compares on standardized llama2-7b 4K context testing vs some Nvidia cards: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3aePczz3zlbJW-Y4/edit#gid=1788227831 https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp... Not on the same platform so hard to make any solid conclusions, but vs the 4090 on a perf/$ basis right now, it's not too bad - about 60% of the inference performance of a 4090 at 45% the price. Still, if ML is your main use case, I think it's hard to argue that a used 3090 wouldn't be the way to go.
- MezzoDelCammin 3y agoyeah, a used 3090 would be the most likely candidate if my next build was something for personal use only. Thing is, I'm probably going to expense it. Which limits me to new components. And I'm not in too much of a rush (weeks / short months). Can afford to wait for a month / two to see how the ROCm support pans out on the AMD side. 60% performance for <50% price is a compelling argument. If the ML support is decent... Though the 4090 is somewhat expensive, I might end up taking it anyway just for the simple reason that it's the only 24GB card with CUDA.
- lhl 3y agoMakes sense to me, although if you're using it professionally as an individual, then I would also seriously consider a last-gen A6000 (about $4K) - despite advances in quants, I'd say 48GB is a better minimum bar atm especially if you want to use long context, run multiple model types, do local fine tuning, etc.
- mijoharas 3y agoCan anyone provide any additional details on the EXL2[0]/GPTQ[1] quantisation, which seems to be the main reason for a speedup in this model? I had a quick look at the paper which is _reasonably_ clear, but if anyone else has any other sources that are easy to understand, or a quick explanation to give more insight into it, I'd appreciate it. [0] https://github.com/turboderp/exllamav2#exl2-quantization https://github.com/turboderp/exllamav2#exl2-quantization [1] https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323
- qeternity 3y ago> which seems to be the main reason for a speedup in this model I wouldn't agree with this as there are substantial perf improvements on GPTQ models of ca. 20%.
- mijoharas 3y agoOk, this article seems to have a nice more approachable explanation[0] [0] https://towardsdatascience.com/4-bit-quantization-with-gptq-36b0f4f02c34 https://towardsdatascience.com/4-bit-quantization-with-gptq-...
- ChatGTP 3y agoUpvoted
- 0xDEF 3y agoIs the LLM local hosting community big enough for Nvidia to release something equivalent to a 4070 but with 40-80 GB of VRAM? A small group of Microsoft Flight Simulator enthusiasts might also like such a GPU.
- janekm 3y agoIt's more that the LLM non-local hosting (and training) community (on A100/H100) is big enough that it's more profitable for Nvidia to not release such prosumer GPUs ;)
- ganoushoreilly 3y agoI'm having the hardest time finding it, but I read a forum post about a few people that were able to add vram to existing 40 series cards after they finally cracked signing for the bios. we're creeping up on the time where more consumer cards will have increases in vram (looks like 40 series supers will have a bump) and within the next couple of years i wouldn't be surprised to see mid priced cards with >= 24gb. Exciting times for sure.
- jmorgan 3y agoThat's fast. It's exciting to see more ways to run these models locally. How does this compare to llama.cpp – both in speed and approach?
- lappa 3y agoInteresting how this method quantizes different layers / modules in a manner that minimizes perplexity as it adjusts parameters. I'd be interested to see how 2.5 bit quantization compares to an unadjusted 4-bit baseline. Additionally would be interesting to see the usual benchmarks (ARC, HellaSwag, MMLU, TruthfulQA) of this method at different average bitrates (2.0, 2.5, 3.0, 4.0, ...). Would also be interesting to see if an average bitrate of 4 is just as fast and small as a constant bitrate of 4, but more accurate. Very exciting work, looking forward to trying this out on my models!
- supermatt 3y agoThis is fantastic. it's really nice to see the different speed comparisons. Does anyone know how a large model performs on some serious hardware? How much faster are these top-grade Nvidia cards than a 4090? Do models that require multiple cards to run in memory run with equivalent performance of the multiple cards or are they throttled by the memory access?
- brucethemoose2 3y ago> Do models that require multiple cards to run in memory run with equivalent performance of the multiple cards or are they throttled by the memory access? It depends, but some frameworks just run the layers sequentially. So you get ~50% utilization from 2 GPUs unless you pipeline requests from multiple users
- jacquesm 3y agoWeird question: has anybody tried what happens if you reduce the models to single bit width? Do they produce complete gibberish or do they still work?
- justincormack 3y agoThere is some work on related areas (pre LLM) eg https://cmu-odml.github.io/papers/XOR-Net_An_Efficient_Computation_Pipeline_for_Binary_Neural_Network_Inference_on_Edge_Devices.pdf https://cmu-odml.github.io/papers/XOR-Net_An_Efficient_Compu...
- jacquesm 3y agoWow, thank you for digging that up, I suspected there might be some gain there but from the abstract it looks like I was off by a massive amount in my naive estimate of what might be achievable. "Theoretical analysis shows that XOR-Net reduces one-third of the bit-wise operations compared with traditional binary convolution, and up to 40% of the full- precision operations compared with XNOR-Net. Experimental results show that our XOR-Net binary convolution without scaling factors achieves up to 135× speedup and consumes no more than 0.8% energy compared with parallel full-precision convolution." The big question is whether or not the results are still as good. It would be super interesting to see whether applying this to LLMs would give comparable benefits.
- atharv_jaju 3y agoHow do you count tokens per second?
- csjh 3y agoYou measure how many tokens are output in a period of time and take the average?
- atharv_jaju 3y agoOh my bad, I meant how to code it?
- csjh 3y agoPython pseudocode: ```py output = prompt("My favourite animal is ") # returns a generator start = time.time() tokens = 0 for token in output: tokens += 1 print(token) print(f"Outputted {tokens / (time.time() - start)} tokens/second") ```
- atharv_jaju 3y agoWhat is the best performing Codellama model on a Macbook with M1 chip? Will Gguf 7Bn 4bit quantized model be good to run locally on the Mac?
- syntaxing 3y agoI would use Ollama. The answer depends on your RAM size. 13B at 4 bit just fits on a 16GB. 34B at 4bit will require 32GB.
- lynguist 3y agoDoes each parameter need double the allocation size?
- syntaxing 3y agoNot really, a 13B 4 bit fits in about 8GB of RAM. But you have to factor in the unified memory aspect of M series chips. Your “vram” and ram are from the same pool. So your computer shares the same pool to run the OS and application hence why you can only load a 13B 4bit for 16GB RAM.
- atharv_jaju 3y agoI have 8GB RAM. So I am able to run 7B 4bit at maximum. I used GGUF + llama.cpp. Will try out Ollama. Thanks for the info!
- Art9681 3y agoWhat is the best model that would run on a 64GB M2 Max? I'm interested in attempting 70B model but not gimped to 2bit if I can avoid it. Would I bet able to run 4bit 70B?
- syntaxing 3y agoThe biggest issue with local LLM is that best is extremely relative. As in, best for what application? If you want a generic LLM that does ok at everything, you can try Vicuna. If you want coding, code llama 34B is really good. Surprisingly, the 13B code up isn’t not as good, but pretty darn close as code llama 34B.
- lynguist 3y agoWhat hardware and software would be recommended for a "good quality" local inference with a LLM on: - a Windows or Linux machine - a Mac?
- rootusrootus 3y agoI think it depends a lot on what you're wanting to do. If you're just playing, you can get surprisingly good results from a 13B model and run it on a fairly basic M1 Mac. And on an M2 Ultra with a lot of RAM you can run some seriously large models at a good speed. I compromised. Was going to build a PC with a nice graphics card just for LLM work, but in the end decided I didn't want the extra hardware and I could be happy with a powerful laptop that is my daily driver but also capable of running a good size LLM. Ended up with an M2 Max w/96GB of RAM. I can run 70B models (quantized at 4 bits) at a usable pace. Not 35 tokens/sec like this 4090 demo, but 5 tok/sec or so, which is usable for me.
- smoldesu 3y agoOn Windows and Linux, you can run 7B models on pretty much any CPU with AVX support. Most x86 CPUs that support SSE can get usable acceleration on Llama.cpp.
- NelsonMinar 3y agoI've been learning a lot by watching the subreddit "Local LLaMA". Here's their discussion of the release: https://www.reddit.com/r/LocalLLaMA/comments/16gq2gu/exllama_v2_has_dropped/ https://www.reddit.com/r/LocalLLaMA/comments/16gq2gu/exllama...
- 3abiton 3y agoUnfortunate that reddit is becoming a close community with their recent api change. I hope things move to Lemmy.
- htsh 3y agoThis subreddit remained open. Unfortunately, however, the oobabooga one went closed for a while and lost a lot of momentum. It is also back, however. Are there good lemmy spaces for LLMs?
- mplewis 3y agoCommunity is a chicken-and-egg problem. I started my own Lemmy server to try to contribute and hold some space for discussion, but if you just want to start a community, it's as easy as making a new account on https://lemmy.world https://lemmy.world and clicking Create a Community on the front page.
- kelvie 3y agoFor localllama this is the most active one so far (which isn't to say it is very active at all): https://sh.itjust.works/c/localllama https://sh.itjust.works/c/localllama If you post topics you'll generally get at least a few responses a day though.
- mlsu 3y agoPardon me for asking a bunch of consumer-gpu questions: 1) is there any architectural difference between the 3090 and the 4090? Are there models that will run on 3 series that will not run on 4 series cards due to feature incompatibility? or is it just raw speed/power efficiency? 2) And; on raw speed. My understanding is that the 4090 is roughly 2x faster than the 3090, right? Are there any things in particular where the 4090 is much much faster (such as, big chip architecture changes that really accelerate some operations)? 3) And somewhat relatedly, if you have a 2x 3090 rig, is that about as fast as a single 4090? 4) And; if you have a 2x 3090 rig, are you able to train larger models? My understanding is that training/finetuning requires significantly more VRAM (and that is because inference can be done with quantization, whereas training must be done at full fp16 precision, right?) But can you train e.g. a model that requires 48gb of space with 2 3090s on the same rig, or do you need a large card with 48gb, like an a4000?
- deleted 3y ago[deleted]
- coolspot 3y ago> Are there models that will run on 3 series that will not run on 4 series cards due to feature incompatibility? No, it is all the same matrix multiplication. > if you have a 2x 3090 rig, is that about as fast as a single 4090? Yes, but double the VRAM > But can you train e.g. a model that requires 48gb of space with 2 3090s on the same rig, or do you need a large card with 48gb, like an a4000? Yes, 2x 3090 would be able to handle anything that a 48Gb card can. For training you will need space for weights and gradients. If using Adam optimizer, that would be 2x-4x model size. Plus weights, plus activations, plus inputs times batch size. So a 24Gb card can train approximately a 3B model without compromises. I am not a lawyer.
- quickthrower2 3y agoI am not your data scientist