Yeah, this is one area we’re struggling with due to the sheer volume of variations and conditions, but I’ve been thinking of collecting some real-time statistic

Yeah, this is one area we’re struggling with due to the sheer volume of variations and conditions, but I’ve been thinking of collecting some real-time statistics around latency, prefill/decode, and model distribution… that way we can update some kind of live + aggregated performance numbers for interested parties.