3 ms·
The only AI use case that cares about latency is interactive voice agents, where you ideally want <200ms response time, and 100ms of network latency kills that.
by CuriouslyC 6mo ago
The only AI use case that cares about latency is interactive voice agents, where you ideally want <200ms response time, and 100ms of network latency kills that. For coding and batch job agents anything under 1s isn't going to matter to the user.
- electroly 6mo agotbh, that's a good point about the voice agents that I hadn't considered. I guess there are some latency-sensitive inference workloads. Thanks for pointing that out.
- devolving-dev 6mo agoYeah, also stuff like robotics which might not really exist today but could be big in the future.
- vanviegen 6mo agoYou'll want the time-sensitive parts (motor control) to be running locally anyway.
- coredog64 6mo agoA customer service chatbot can require more than one LLM call per response to the point that latency anywhere in the system starts to show up as a degraded end-user experience.