3 ms·
Having used groq and other fast LLM services a fair bit, lag seems negligible. You're literally just passing text at close to the speed of light.
by whimsicalism 3y ago
Having used groq and other fast LLM services a fair bit, lag seems negligible. You're literally just passing text at close to the speed of light.
- jonplackett 3y ago* when you have a good internet connection ** when you live in the USA
- whimsicalism 3y ago> * when you have a good internet connection Or at least, a good enough internet connection to send plaintext. > * when you live in the USA Even from Australia to USA is just ~300ms of latency for first token and then the whole thing can finish in ~1s. And making that faster doesn't require on-device deployment, it just requires a server in Australia - which is obviously going to be coming if it hasn't already for many providers.