3 ms·
Amazing! I use text-generation-webui to play with LLMs, but was always jealous of this much simpler interface. Somewhat related note -- does anyone know what a
by kelvie 3y ago
Amazing! I use text-generation-webui to play with LLMs, but was always jealous of this much simpler interface.
Somewhat related note -- does anyone know what are the performance differences for GPU-only inference using this loader (llama.cpp + GGUF/GGML modles) vs exllama using GPTQ? My understanding is that exllama/GPTQ gets a lot higher tok/s on a consumer GPU like a [34]090.
Would save me many gigabytes of downloads of testing if someone knew.
- qeternity 3y agoIt is hardware and use case dependent but I would say roughly that ExLlama is 10-20% faster than llama.cpp and ExLlama v2 is 10-20% faster than ExLlama (my experiences at 4 bit quantization).
- lhl 3y agoThe numbers are always changing, but from my testing, they're close enough that it doesn't really matter. My most recent benchmarks: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3aePczz3zlbJW-Y4/edit#gid=1788227831 https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp... I'd say that you should pick the backend that has the quantized models or other features (sampler, context window, API compatibility, etc) that suits you best.
- brucethemoose2 3y agoexLLAMAv2 and GPTQ are different implementations, and currently exLLAMA is definitely the best for discrete Nvidia (and AMD?) GPUs. Its faster and uses less VRAM for the same perplexity than pretty much anything else. But the EX2 quantization is very new, and you will have to quantize many models yourself. But its missing some killer features of llama.cpp, like grammar based sampling.