3 ms·
You can use tenacity for retries and wouldn't you want to cache the request / response around the endpoint instead of the gpt call -> that's what we ended up do
by detente18 3y ago
You can use tenacity for retries and wouldn't you want to cache the request / response around the endpoint instead of the gpt call -> that's what we ended up doing.
Streaming output and function-calling support is interesting
- d4rkp4ttern 3y agoCould you elaborate on this — “wouldn’t want to cache around the endpoint instead of GPT call”. Just want to see if I’m missing an important consideration here
- detente18 3y agoYou're probably calling openai from a backend server. If you're making the call because a user asked a question -> your endpoint received it -> did some processing -> called openai -> returned the response, why would you cache the openai call instead of the endpoint that's receiving the user call?