2 ms·
What GPU was used when testing this? Is this faster than HuggingFace's Text Generation inference container?
by ilaksh 3y ago
What GPU was used when testing this?
Is this faster than HuggingFace's Text Generation inference container?
- chillee 3y agoWe used an A100-80GB GPU. We didn't compare explicitly to Huggingface TGI but I think you should be able to compare the tokens/s achieved. One note is that this release is optimized for latency, while I think HF TGI might be more optimized for throughput.