3 ms·
Falcon 7B/40B is good, but hasn't really caught on like LLaMA has. One big reason is that its not well supported by 4-bit inference code (namely llama.cpp and G
by brucethemoose2 3y ago
Falcon 7B/40B is good, but hasn't really caught on like LLaMA has. One big reason is that its not well supported by 4-bit inference code (namely llama.cpp and GPTQ).
Mosaic's 7B (and 30B?) models have the same issue, and 7B kinda paled in comparison to LLaMA 7B... But maybe it would be better if finetuned?
To me, its kinda baffling that Mosaic didn't work on adding highly quantized inference to the popular frameworks.
- Tostino 3y agoFalcon is also slow for inference compared to LLaMA models of similar size. Speed can turn into a quality all its own when you break up problems into small pieces and use the LLM to iterate.
- brucethemoose2 3y agoThats because its unoptimized, right? My impression is that the Falcon GPTQ (and 8 bit bnb inference?) code is just immature.
- ml_hardware 3y agoMosaic's MPT models are already supported in GGML: https://github.com/ggerganov/ggml https://github.com/ggerganov/ggml Here's MPT-30B running in 4-bit precision on CPU :) https://twitter.com/abacaj/status/1673133443339763712?s=20 https://twitter.com/abacaj/status/1673133443339763712?s=20
- brucethemoose2 3y agoOh I didn't realize this. Everything is moving so fast that I can't even keep up with the features.