4 ms·
The obvious burning question is how performance looks over different network conditions on some standard models. Have you done much benchmarking? Is it mainly l
by zmmmmm 3mo ago
The obvious burning question is how performance looks over different network conditions on some standard models. Have you done much benchmarking? Is it mainly latency affected or is overall throughput less than the capacity of the GPUs due to being distributed?
- sig_kill 3mo agoYeah, this is one area we’re struggling with due to the sheer volume of variations and conditions, but I’ve been thinking of collecting some real-time statistics around latency, prefill/decode, and model distribution… that way we can update some kind of live + aggregated performance numbers for interested parties.