3 ms·
How fast is it in single batch mode?
by ImprobableTruth 4y ago
How fast is it in single batch mode?
- Miraste 4y agoAfter turning on compression I was able to fit the whole thing in GPU memory and then it became much faster. Not ChatGPT speeds or anything, but under a minute for a response in their chatbot demo. A few seconds in some cases.