4 ms·
It seems like they're caching information/replies on key words which is a good optimization.
by Falimonda 2y ago
It seems like they're caching information/replies on key words which is a good optimization.
- iknownthing 2y agoI assumed it was coming straight from the API because of the token-by-token generation effect but maybe you're right.
- rodiger 2y agoThe token-by-token responses are probably API, while the "instant" loads seem to be cached. Probably using groq based on speed of response