3 ms·
That is bs=32 which is wholly unimpressive. Last gen consumer cards can do better than this at similar power envelope. Current gen even better.
by qeternity 2y ago
That is bs=32 which is wholly unimpressive. Last gen consumer cards can do better than this at similar power envelope. Current gen even better.
- inhumantsar 2y agodo you have any benchmarks to share? the best I've seen for a 4090 are all in the mid-20s, never more than 30t/s.
- qeternity 2y agoWhere have you seen this? Look at what vLLM or TensorRT will do on a 4090 at those batch sizes.
- deleted 2y ago[deleted]