3 ms·
Do you build agents that interface with web browsers? BLAST is sort of like vllm for browser+LLM. The motivation for this is that browser+LLM is slow and we can
by calebhwin 1y ago
Do you build agents that interface with web browsers? BLAST is sort of like vllm for browser+LLM. The motivation for this is that browser+LLM is slow and we can do a lot of optimization with an engine that manages browser+LLM together - e.g. prefix caching, auto-parallelism, data parallelism, request hedging, scheduling policy, and more coming soon.
Now the API is what may be throwing folks off. Right now it's an OpenAI-compatible API. We will implement MCP. But really the core thing is abstracting away optimizations required to efficiently run browser+LLM.
- otabdeveloper4 1y agoI have no idea what you mean by "browser+LLM". Image models to process pictures of a webpage? A wrapper around Python's "requests"?
- calebhwin 1y agoA system that does the following given a task_description: while LLM("is <task_description> not done?"): Browser.run(LLM("what should the browser do next given <Browser.get_state()>")) This simple loop turns out to be very powerful, achieving the highest performance on some of OpenAI's latest benchmarks. But it's also heavily unoptimized compared to a system that is just LLM("<task_description>") for which we already have things like vllm. BLAST is a first step towards optimizing this while loop.