4 ms·
For those suffering from deceptive graph fatigue, this is impressive. exLlama is blazing fast. Even if they just benched exllamav1, exllamav2 is only a bit fas
by brucethemoose2 3y ago
For those suffering from deceptive graph fatigue, this is impressive.
exLlama is blazing fast. Even if they just benched exllamav1, exllamav2 is only a bit faster, at least on my single 3090 in a similar environment.
vLLM is focused more on batching performance, but even then MLC/TVM looks like its putting up a fight without batching.
I am a bit fatigued with llama backends myself, and it looks like this won't help me run 70B in a single 3090, but I need to dig into mlc again.
- junrushao1994 3y agoYeah thanks for sharing! This is definitely super valuable data and insights :) Regarding exllama-V2, MLC/TVM does benchmark against it: - Single GPU: https://github.com/mlc-ai/llm-perf-bench#int4-quantized-single-gpu https://github.com/mlc-ai/llm-perf-bench#int4-quantized-sing... - Multi GPU: Figure 2 in the blog: http://blog.mlc.ai/2023/10/19/Scalable-Language-Model-Inference-on-Multiple-NVDIA-AMD-GPUs#scalability http://blog.mlc.ai/2023/10/19/Scalable-Language-Model-Infere... > vLLM focuses more on batching performance Exactly. vLLM doesn’t optimize for latency-first scenarios as it focuses on throughput, i.e. batching. This particular blog post instead focuses particular on latency, i.e. the fastest you could possible get with those many GPUsz Regarding batching, it is coming pretty soon, and we will have another blog post on this.