47 ms·
> Imagine a conversational interface with much less latency With current models the latency comes from processing, not from the network — going from a high-pow
by Shish2k 2y ago
> Imagine a conversational interface with much less latency
With current models the latency comes from processing, not from the network — going from a high-power remote server to a low-power local phone is likely to increase latency more than it reduces it
- ec109685 2y agoIt depends on how big the model is. They are using a little LLM to help correct text and that would be dreadful if it took a server round trip.