4 ms·
resource contention under high concurrency A queue? What else can you really do. Your server is at the mercy of OpenAI, so all you can do is queue up everyone'
by ivape 1y ago
resource contention under high concurrency
A queue? What else can you really do. Your server is at the mercy of OpenAI, so all you can do is queue up everyone's requests. I don't know how many parallel requests you can send out to OpenAI (infinite?), so that bottleneck is probably just dependent on your server stack (how many threads).
There's a lot of language being thrown out here, and I'm trying to see if we're using too much language to discuss basic concepts.
- calebhwin 1y agoThere's definitely opportunities to parallelize. BLAST exploits these with an LLM-planner and tool calls to dynamically spawn/join subtasks (there's also data parallelism and request hedging which further reduce latency). Now you are right that at some point you'll get throttled either by LLM rate limits or a set budget for browser memory usage or LLM cost. BLAST's scheduler is aware of these constraints and uses them to effectively map tasks to resources (resource=browser+LLM).