3 ms·
Everyone compares against llama.cpp because it's easy mode. Llama.cpp is slow! Everyone should know this. They should compare against exllamav2 or other optimiz
by modeless 3y ago
Everyone compares against llama.cpp because it's easy mode. Llama.cpp is slow! Everyone should know this. They should compare against exllamav2 or other optimized implementations.
- nulld3v 3y agoExLlama is GPU only right? This speedup is for GPU + CPU split use cases.
- modeless 3y agoOh I see, they are running a 40B model unquantized, whereas exllamav2 would have to use 4-bit quantization to fit. Given the quality of 4-bit quantization these days and the speed boost it provides I question the utility of running unquantized for serving purposes. I see they have a 4-bit benchmark lower down in the page. That's where they ought to compare against exllamav2.
- sroussey 3y agoWhat do you recommend that is faster that I can package into an app for distribution?
- modeless 3y agoI have packaged exllamav2 (plus a lot of other stuff) into an app for distribution here: https://apps.microsoft.com/detail/9NC624PBFGB7 https://apps.microsoft.com/detail/9NC624PBFGB7 I used pyinstaller. It was difficult because Python makes these things difficult. But it works. It does require an Nvidia GPU. MLC-LLM is another option that might be easier to package and potentially able to run on AMD.
- sroussey 3y agoOh yeah, I want to work on AMD/Intel/NVIDIA and MacOS, even iOS/Android. I've been following MLC-LLM as well. Right now I am just using JS/WASM from Huggingface, but later I will want something more performant.
- modeless 3y agoYeah if you want maximum performance on multiple platforms you'll probably have to package multiple frameworks. Llama.cpp might be a decently fast option on Apple Silicon, I'm not sure of the state of the art there.
- avereveard 3y agoYeah but exllama doesn't do grammars so I'm stuck with llama.cpp Also apparently exllama has a few side effects in coherence https://www.reddit.com/r/LocalLLaMA/comments/17w57eu/llm_format_comparisonbenchmark_70b_gguf_vs_exl2/ https://www.reddit.com/r/LocalLLaMA/comments/17w57eu/llm_for...
- superkuh 3y agoIn this case they're comparing against llama.cpp because the code is literally a modification of llama.cpp. I'm not talking about using the ggml lib for matrix calculations, it's literally using the llama.cpp main.cpp and other normal llama.cpp code. It's a fork. It is directly comparable. https://github.com/ggerganov/llama.cpp/pull/4543 https://github.com/ggerganov/llama.cpp/pull/4543 [Review] Merge PowerInfer with llama.cpp mainline #4543 https://github.com/ggerganov/llama.cpp/discussions/4534#discussioncomment-7900305 https://github.com/ggerganov/llama.cpp/discussions/4534#disc... "The x11 speedup is kind of cherrypicked because the llama.cpp GPU code for Falcon 40b is just not well-optimized."
- modeless 3y agoThanks for pointing that out, I didn't notice that. That makes sense. I still think a comparison with exllamav2 or other optimized inference library would make sense too.