3 ms·
Throughput is a metric for the total number of tokens/sec for all users in the system. Latency (ITL,TTFT) are individual user metrics.
by tipsytoad 7mo ago
Throughput is a metric for the total number of tokens/sec for all users in the system. Latency (ITL,TTFT) are individual user metrics.