2 ms·
But usually AWQ get recommended for GPU inference over GGUF
by 3abiton 3y ago
But usually AWQ get recommended for GPU inference over GGUF
- Mathnerd314 3y agoBy who? Only comparison I have seen is that it sucks vs. EXL2 https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacpp/ https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacp...
- 3abiton 3y agoThe issue is benchmarks for LLMs or models formats are tough to compare, as there are many factors at play. But beyond ooba's comparison, many other sources recommend GPTQ or AWQ for GPU inference as it gives better quality for the same quant level (AWQ apparently takes more VRAM though, but better quality). Given how many models are available I would take these tests with a grain of salts.
- Eisenstein 3y agoHaven't heard this. Was this a few months ago? A lot happens in this space over that time span.