3 ms·

I hope some of the opensource inference servers start supporting that endpoint soon. I know vLLM has added some "offline batch mode" support with the same forma

• asaddhamani 2y ago
  Do note it can take up to 24 hours or drop requests altogether. But if that’s not an issue for your use case it’s a great cost saving.
  • jumploops 2y ago
    This is neat, I’ve been looking for a way to run our analytics (LLM-based) without affecting the rate limits of our prod app. May need to give this a try!
  • altdataseller 2y ago
    What percentage of requests usually get dropped? Is it something miniscule like 1% or are we talking non trivial like 10%
• johndough 2y ago
  llama.cpp enabled continuous batching by default half a year ago: https://github.com/ggerganov/llama.cpp/pull/6231 https://github.com/ggerganov/llama.cpp/pull/6231 There is no need for a new API endpoint. Just send multiple requests at once.
  • Tostino 2y ago
    The point of the endpoint is to be able to standardize my codebase and have an agnostic LLM provider that works the same. Continuous batching is helpful for this type of thing, but it really isn't everything you need. You'd ideally maintain a low priority queue for the batch endpoint and a high priority queue for your real-time chat/completions endpoint. Would allow utilizing your hardware much better.