28 ms·
How does this specifically compare to llama.cpp, vllm and oobabooga? All of them already have openapi compatible API with some caveats.
by lxe 3y ago
How does this specifically compare to llama.cpp, vllm and oobabooga? All of them already have openapi compatible API with some caveats.
- bbor 3y agoNot an author but the answer seems to involve their espoused focus on ease-of-use and “it-just-works” levels of modularity. See their list of plans, which include a few LLMs and a variety of adjacent tools/features. It’s a basic value prop and I’m way behind the times on local inference (so barely a few months lol), but it seems different enough that I’m excited to try it. Other tools have tried this path but I’ve seen none with a demo page quite this convincing…
- p-e-w 3y agoOoba is a monstrosity with gigabytes of dependencies. I love it and use it daily, but if all you need is a compatible API, it's massive overkill.