3 ms·

LoRAX [0] does sound super helpful and so I’d be curious if there are some good examples of people applying it. What are some current working deployments where

LoRAX [0] does sound super helpful and so I’d be curious if there are some good examples of people applying it. What are some current working deployments where one has 100s or 1000s of LoRA fine tuned models? I guess I can make up stuff that makes sense, so that’s not really what I’m asking, I’m interested in learning about any known deployments and example setups. [0] https://github.com/predibase/lorax https://github.com/predibase/lorax

brucethemoose2 3y ago
There aren't really any I know of, because its brand new and everyone just uses vllm :P No one knows about it! Which is ridiculous because batched requests with loras is mind blowing! Just like many other awesome backends like InternLM's backend, LiteLLM, Outline's VLLM fork, Aphroidte, exllamav2 batching servers and and such. Heck, a lot of trainers don't even publish the loras they merge into base models. Personally we are waiting on the integration with constrained grammar before swapping to Lorax. Then I am going to add exl2 quantization support myself... I hope.
- semmulder 3y ago
  FYI, vLLM also just added experimental multi-lora support: https://github.com/vllm-project/vllm/releases/tag/v0.3.0 https://github.com/vllm-project/vllm/releases/tag/v0.3.0 Also check out the new prefix caching, I see huge potential for batch processing purposes there!
  - brucethemoose2 3y ago
    Missed this, thanks. Everything is moving so fast!
  - tgaddair 3y ago
    Yes, we (LoRAX devs) saw that (we know the author pretty well). It's a useful addition, though quite a bit simpler than our level of support for multi-LoRA inference. We're planning on doing a more comprehensive comparison soon, now that it's officially out. I will say that if you want to explore the forefront of this multi-LoRA inference, definitely worth giving LoRAX a look. We just added support for per-request model merging (https://predibase.github.io/lorax/guides/merging_adapters/ https://predibase.github.io/lorax/guides/merging_adapters/) as an example, and are planning on continuing to double down on this idea of combining adapters in some pretty unique ways.
- tgaddair 3y ago
  Thanks for the shoutout! We are adding the Outlines integration to LoRAX in the next two weeks, so stay tuned!

brucethemoose2 3y ago

There aren't really any I know of, because its brand new and everyone just uses vllm :P No one knows about it! Which is ridiculous because batched requests with loras is mind blowing! Just like many other awesome backends like InternLM's backend, LiteLLM, Outline's VLLM fork, Aphroidte, exllamav2 batching servers and and such. Heck, a lot of trainers don't even publish the loras they merge into base models. Personally we are waiting on the integration with constrained grammar before swapping to Lorax. Then I am going to add exl2 quantization support myself... I hope.

semmulder 3y ago
FYI, vLLM also just added experimental multi-lora support: https://github.com/vllm-project/vllm/releases/tag/v0.3.0 https://github.com/vllm-project/vllm/releases/tag/v0.3.0 Also check out the new prefix caching, I see huge potential for batch processing purposes there!
- brucethemoose2 3y ago
  Missed this, thanks. Everything is moving so fast!
- tgaddair 3y ago
  Yes, we (LoRAX devs) saw that (we know the author pretty well). It's a useful addition, though quite a bit simpler than our level of support for multi-LoRA inference. We're planning on doing a more comprehensive comparison soon, now that it's officially out. I will say that if you want to explore the forefront of this multi-LoRA inference, definitely worth giving LoRAX a look. We just added support for per-request model merging (https://predibase.github.io/lorax/guides/merging_adapters/ https://predibase.github.io/lorax/guides/merging_adapters/) as an example, and are planning on continuing to double down on this idea of combining adapters in some pretty unique ways.
tgaddair 3y ago
Thanks for the shoutout! We are adding the Outlines integration to LoRAX in the next two weeks, so stay tuned!