3 ms·

what makes it slow? is it because they throttle your api key?

• itake 3y ago
  Our p99 for gpt4 is 3s. Images take up to 50s.
  • spxneo 3y ago
    so how would you go about improving that?
    • freediver 3y ago
      Not using an LLM for it.
    • itake 3y ago
      we only send 0.5-5% of traffic to gpt4, thanks to smaller faster cheaper models. So not all of our traffic is hit with 50s latencies :-/
• tuckerconnelly 3y ago
  Chain of thought takes time to generate all the characters. If you do a chain-of-thought for every action and every misstep (and you need to for quality + reliability), it adds up.
  • spxneo 3y ago
    Is there no way to share that "memory" across chats? or are we at the mercy of hosted models?
    • ShamelessC 3y ago
      There’s caching but only so much can be cached when small changes in the input can lead to an entirely different space of outputs. Furthermore, even with caching LLM inference can take anywhere from 1-15s using GPT4-Turbo via the API. As was mentioned, the more characters you prefix in the context - the longer this takes. Similarly you have a variable length output from model (up to a fixed context length) and so the time it takes to calculate the “answer” can also take awhile. In particular with CoT you are basically forcing the model to use more characters than it otherwise would (in its answer) by asking it to explain itself in a verbose step by step manner.