3 ms·
It is hardware and use case dependent but I would say roughly that ExLlama is 10-20% faster than llama.cpp and ExLlama v2 is 10-20% faster than ExLlama (my expe
by qeternity 3y ago
It is hardware and use case dependent but I would say roughly that ExLlama is 10-20% faster than llama.cpp and ExLlama v2 is 10-20% faster than ExLlama (my experiences at 4 bit quantization).