5 ms·
This . Path of least resistance wins. At the end if the day it's easier to get an oauth token and quota from OpenAI than spend time setting things up locally an
by kuchenbecker 3y ago
This . Path of least resistance wins. At the end if the day it's easier to get an oauth token and quota from OpenAI than spend time setting things up locally and get inferior results.
I do have llama 2 running locally but for fun but in business I buy unless it's a core competency. You'd burn millions per year trying to match off the shelf just in eng costs.
- visarga 3y agoIn practice the OpenAI-GPT and Azure-GPT are severely rate limited such that they err out one in 10 attempts, impose timeouts, and most importantly, require you trust them with your data.
- phillipcarter 3y agoThat’s really not the case at all. In our experience in production there’s no timeouts and latency has only gotten better over time. Your comment was more accurate in early May of this year (but even then an exaggeration).
- visarga 3y agoJust yesterday I was writing a test function with 4 cases, each sending one call to chatGPT. I didn't have delays between them, they were running back-to-back. After 1-2 attempts it started complaining of API limits and advised me to wait 51 seconds. How can I make a test like that? It raises exceptions for reasons of Azure congestion, not task error.
- phillipcarter 3y agoSounds like you're hitting rate limiting for tokens per minute. I'd suggest working on reducing the size of your prompt, which will also result in more accurate responses (since LLMs do poorly when prompts get past a certain size).