2 ms·
It says 8 A100 for normal inference. How different is the 4bit performance?
by ilaksh 3y ago
It says 8 A100 for normal inference. How different is the 4bit performance?
- Tostino 3y agoNot that I've tested it with this model, but I have for Llama2 models...it makes a minimal difference. I haven't found anything that was noticeable with ~30-70b models, and from what I can tell from the scaling charts, it matters even less with the larger parameter count models.