3 ms·
There aren't really any I know of, because its brand new and everyone just uses vllm :P No one knows about it! Which is ridiculous because batched requests wit
by brucethemoose2 3y ago
There aren't really any I know of, because its brand new and everyone just uses vllm :P
No one knows about it! Which is ridiculous because batched requests with loras is mind blowing! Just like many other awesome backends like InternLM's backend, LiteLLM, Outline's VLLM fork, Aphroidte, exllamav2 batching servers and and such. Heck, a lot of trainers don't even publish the loras they merge into base models.
Personally we are waiting on the integration with constrained grammar before swapping to Lorax. Then I am going to add exl2 quantization support myself... I hope.
- semmulder 3y agoFYI, vLLM also just added experimental multi-lora support: https://github.com/vllm-project/vllm/releases/tag/v0.3.0 https://github.com/vllm-project/vllm/releases/tag/v0.3.0 Also check out the new prefix caching, I see huge potential for batch processing purposes there!
- brucethemoose2 3y agoMissed this, thanks. Everything is moving so fast!
- tgaddair 3y agoYes, we (LoRAX devs) saw that (we know the author pretty well). It's a useful addition, though quite a bit simpler than our level of support for multi-LoRA inference. We're planning on doing a more comprehensive comparison soon, now that it's officially out. I will say that if you want to explore the forefront of this multi-LoRA inference, definitely worth giving LoRAX a look. We just added support for per-request model merging (https://predibase.github.io/lorax/guides/merging_adapters/ https://predibase.github.io/lorax/guides/merging_adapters/) as an example, and are planning on continuing to double down on this idea of combining adapters in some pretty unique ways.
- tgaddair 3y agoThanks for the shoutout! We are adding the Outlines integration to LoRAX in the next two weeks, so stay tuned!