14 ms·
How does BLAST handle browser instance isolation and resource contention under high concurrency?
by badmonster 1y ago
How does BLAST handle browser instance isolation and resource contention under high concurrency?
- ivape 1y agoresource contention under high concurrency A queue? What else can you really do. Your server is at the mercy of OpenAI, so all you can do is queue up everyone's requests. I don't know how many parallel requests you can send out to OpenAI (infinite?), so that bottleneck is probably just dependent on your server stack (how many threads). There's a lot of language being thrown out here, and I'm trying to see if we're using too much language to discuss basic concepts.
- calebhwin 1y agoThere's definitely opportunities to parallelize. BLAST exploits these with an LLM-planner and tool calls to dynamically spawn/join subtasks (there's also data parallelism and request hedging which further reduce latency). Now you are right that at some point you'll get throttled either by LLM rate limits or a set budget for browser memory usage or LLM cost. BLAST's scheduler is aware of these constraints and uses them to effectively map tasks to resources (resource=browser+LLM).