tgaddair

4 ms·

1. 1.▲
   by tgaddair 3y ago

   In LoRAX v0.8 we've added native integration with Outlines, allowing you to guarantee your output always comes back in the structure of your choosing. But while structured generation can guarantee the right format comes back, it cannot
2. 2.▲
   LoRAX and Outlines: Better JSON Extraction with Structured Generation and LoRA(predibase.com)

   7 points by tgaddair 3y ago|2 comments
3. 3.▲
   by tgaddair 3y ago

   Yes, we (LoRAX devs) saw that (we know the author pretty well). It's a useful addition, though quite a bit simpler than our level of support for multi-LoRA inference. We're planning on doing a more comprehensive comparison soon, n
4. 4.▲
   by tgaddair 3y ago

   Thanks for the shoutout! We are adding the Outlines integration to LoRAX in the next two weeks, so stay tuned!
5. 5.▲
   by tgaddair 3y ago

   Hey, LoRAX dev here. This was one thing we spent a lot of effort optimizing. The TL;DR is that in most cases latency will be with 80% of the baseline latency with 0 adapters with as many as 128 adapters at once under heavy request load. Che