> On-device inference improves not just its latency this is a nuanced thing. sure you save the network hop, but you are running inference on a much more underp

> On-device inference improves not just its latency this is a nuanced thing. sure you save the network hop, but you are running inference on a much more underpowered model on an underpowered device. at the end of the day you want to get the lowest time to first token and highest tokens per second. on-device optimizes TTFT at the expense of Tok/s. i think there are many usecases where Tok/s dominates in overall latency considerations.