3 ms·
In practice the OpenAI-GPT and Azure-GPT are severely rate limited such that they err out one in 10 attempts, impose timeouts, and most importantly, require you
by visarga 3y ago
In practice the OpenAI-GPT and Azure-GPT are severely rate limited such that they err out one in 10 attempts, impose timeouts, and most importantly, require you trust them with your data.
- phillipcarter 3y agoThat’s really not the case at all. In our experience in production there’s no timeouts and latency has only gotten better over time. Your comment was more accurate in early May of this year (but even then an exaggeration).
- visarga 3y agoJust yesterday I was writing a test function with 4 cases, each sending one call to chatGPT. I didn't have delays between them, they were running back-to-back. After 1-2 attempts it started complaining of API limits and advised me to wait 51 seconds. How can I make a test like that? It raises exceptions for reasons of Azure congestion, not task error.
- phillipcarter 3y agoSounds like you're hitting rate limiting for tokens per minute. I'd suggest working on reducing the size of your prompt, which will also result in more accurate responses (since LLMs do poorly when prompts get past a certain size).