4 ms·
FWIW, we've been live with GPT-3.5 turbo since May and it's improved a LOT. Latency is down consistently across the board and we haven't seen a single "429 mod
by phillipcarter 3y ago
FWIW, we've been live with GPT-3.5 turbo since May and it's improved a LOT.
Latency is down consistently across the board and we haven't seen a single "429 model overloaded" error in the past month: https://twitter.com/_cartermp/status/1686894576202907651/ https://twitter.com/_cartermp/status/1686894576202907651/
- waffletower 3y agoIt is worth noting that I had the perception, incorrect or not, that larger customers were being given priority to OpenAI APIs. It is good to see that you are reporting improvements.
- phillipcarter 3y agoOh, they undoubtedly were. We have decent volume, but worked very hard in our prompting to minimize input and output tokens, so we don't have a big bill. I just think that in the past months they've added significantly more capacity and done a lot of work on their inference servers to make things run better.
- waffletower 3y agoGiven Meta's obvious, and most welcome, open source play with LLaMA 2, it behooves OpenAI to be performant and accessible to everyone. For my application, I am finding that 70b LLaMA 2 models are very close to the inference quality of gpt-3.5-turbo.
- hooande 3y agoIt's gotten better for everyone in the last few months. It used to be a nightmare, but I haven't seen a timeout or rate limit error in a long time.