3 ms·
That’s really not the case at all. In our experience in production there’s no timeouts and latency has only gotten better over time. Your comment was more accur
by phillipcarter 3y ago
That’s really not the case at all. In our experience in production there’s no timeouts and latency has only gotten better over time. Your comment was more accurate in early May of this year (but even then an exaggeration).
- visarga 3y agoJust yesterday I was writing a test function with 4 cases, each sending one call to chatGPT. I didn't have delays between them, they were running back-to-back. After 1-2 attempts it started complaining of API limits and advised me to wait 51 seconds. How can I make a test like that? It raises exceptions for reasons of Azure congestion, not task error.
- phillipcarter 3y agoSounds like you're hitting rate limiting for tokens per minute. I'd suggest working on reducing the size of your prompt, which will also result in more accurate responses (since LLMs do poorly when prompts get past a certain size).