4 ms·
do you have any benchmarks to share? the best I've seen for a 4090 are all in the mid-20s, never more than 30t/s.
by inhumantsar 2y ago
do you have any benchmarks to share? the best I've seen for a 4090 are all in the mid-20s, never more than 30t/s.
- qeternity 2y agoWhere have you seen this? Look at what vLLM or TensorRT will do on a 4090 at those batch sizes.
- deleted 2y ago[deleted]