3 ms·

Great start! Are you planning to add to following: Retries w exponential backoff, Caching, Streaming output, Function-calling support

• detente18 3y ago
  You can use tenacity for retries and wouldn't you want to cache the request / response around the endpoint instead of the gpt call -> that's what we ended up doing. Streaming output and function-calling support is interesting
  • d4rkp4ttern 3y ago
    Could you elaborate on this — “wouldn’t want to cache around the endpoint instead of GPT call”. Just want to see if I’m missing an important consideration here
    • detente18 3y ago
      You're probably calling openai from a backend server. If you're making the call because a user asked a question -> your endpoint received it -> did some processing -> called openai -> returned the response, why would you cache the openai call instead of the endpoint that's receiving the user call?
• ij23 3y ago
  good points, probably going to add streaming output, function calling support. As for retries tenacity does a great job already