5 ms·
Falcon is also slow for inference compared to LLaMA models of similar size. Speed can turn into a quality all its own when you break up problems into small piec
by Tostino 3y ago
Falcon is also slow for inference compared to LLaMA models of similar size. Speed can turn into a quality all its own when you break up problems into small pieces and use the LLM to iterate.
- brucethemoose2 3y agoThats because its unoptimized, right? My impression is that the Falcon GPTQ (and 8 bit bnb inference?) code is just immature.