3 ms·
Isn’t Falcon too slow at inference tho?
by behnamoh 3y ago
Isn’t Falcon too slow at inference tho?
- brucethemoose2 3y agoThe GPTQ implementation is reasonably fast, I think. Or maybe they are running it at 8 bits? I just saw that they funded this: https://huggingface.co/ehartford/WizardLM-Uncensored-Falcon-40b https://huggingface.co/ehartford/WizardLM-Uncensored-Falcon-...
- behnamoh 3y agoInteresting! I don’t know what they’re up to and what’s the purpose of their project. But using an uncensored model is kinda risky in production, unless they know what they’re doing.