4 ms·

Woww, very creative and interesting idea: I understand it as: train a bunch of fact-based LoRAs to zero loss (they mention 100k different ones), then use RAG to

Woww, very creative and interesting idea: I understand it as: train a bunch of fact-based LoRAs to zero loss (they mention 100k different ones), then use RAG to pick the appropriate Loras for a query. So cool. The only moat I can think of for such a company would be proprietary fact loras- basically licensing a modern ai encyclopedia. Anyway, really nice idea.

liuliu 2y ago
I think there is an expert router layer to decide which loras to be integrated at inference time. But they also mention that they freeze the weights for router during training. So it is unclear to me how the router was trained on what loss.
- vessenes 2y ago
  Interesting. That’s kind of surprising to me - it would mean with every new Lora they’d need to fine tune the router, no? Embedding a description of the Lora and using RAG to pull the nearest Loras in the embedding space is where my mind goes; it’s super extensible, minimal additional training for customer use cases, and the way the Loras probably work it’s not terrible to pull a few extras. Anyway I just speculate —- no idea what they’re actually doing on the backend.
  - liuliu 2y ago
    That's where it is confusing to me. They mentioned that for LoRA fine-tuning, the router weights are frozen, so you don't update the routing when training different concept. But how that expert router is trained? Could be a pretraining with some aux loss to encourage diversity.
- badriprof 2y ago
  I am old enough to remember the use of yellow pages to find teh right expert
sage92 2y ago
They mention "tuning millions of expert adapters", not 100k
deleted 2y ago
[deleted]
magicalhippo 2y ago
Bit late to the party and I'm not into the AI scene, but from glossing over the three key papers they cite to describe their model, my take is as follows. The idea from LoRA[1] is to take pre-trained, dense weights W_0 for a model and adapt them to new training data by using the weights W = W_0 + BA for inference. The key is that the matrices A and B have a very low rank[2] compared to W_0. Training is then done by treating W_0 as constant, and only updating the A and B matrices. The idea from MoE[3] is to pick just a few "experts" from a large number using Softmax, and insert them between other layers in a neural network. The "experts" can be just some simple matrices or neural networks in their own right. The Lamini model seems to combine these ideas, where they use several "experts" layered between the BA matrices. However instead of just a simple Softmax to select the "experts" they use (chunked?) cross-attention[4], and a much larger number of "experts" compared to the MoE paper. From what I gather the cross-attention allows the expert selection to react to the context, unlike the more naive plain Softmax gating approach. Similarly to LoRA, they train the Lamini model by keeping the pre-trained LLM weights constant. From what I can gather they do a bit of training on the cross-attention layer but then freeze that too, to avoid the cross-attention layer favoring the same "experts" all the time. The idea then is to train the "experts" and the LoRA layer on facts until the combined model (W above) gets each fact correct (zero loss). Thus when the model "sees" a keyword in a sentence, a set of "experts" will steer/adjust the output of the pre-trained LLM to output the correct fact. Or at least that's how I imagine it works. What's less clear to me is what exactly the "experts" are and how they are combined. Since they're used to adjust the weights of the combined model, I it makes the most sense to my un-trained eye that they're simple matrices as mentioned in the MoE paper. Given they're layered between the low-rank portions of the LoRA section, they're necessarily relatively small matrices, so having millions of these "expert" matrices doesn't add too many parameters overall. And while they're portrayed as stacked in the Lamini paper, suggesting matrix multiplication, matrices are generally not commutative[5]. So to the un-trained eye it seems likely the "experts" are simply added like in the MoE paper. But yeah, I'm very much not an expert and the paper was more like a conference poster at best, lacking a lot of detail, so this might be all gibberish and I'd appreciate being corrected. [1]: https://arxiv.org/abs/2106.09685 https://arxiv.org/abs/2106.09685 [2]: https://en.wikipedia.org/wiki/Rank_factorization https://en.wikipedia.org/wiki/Rank_factorization [3]: https://arxiv.org/abs/1701.06538 https://arxiv.org/abs/1701.06538 [4]: https://arxiv.org/abs/2112.04426 https://arxiv.org/abs/2112.04426 [5]: https://en.wikipedia.org/wiki/Commuting_matrices https://en.wikipedia.org/wiki/Commuting_matrices

liuliu 2y ago

I think there is an expert router layer to decide which loras to be integrated at inference time. But they also mention that they freeze the weights for router during training. So it is unclear to me how the router was trained on what loss.

vessenes 2y ago
Interesting. That’s kind of surprising to me - it would mean with every new Lora they’d need to fine tune the router, no? Embedding a description of the Lora and using RAG to pull the nearest Loras in the embedding space is where my mind goes; it’s super extensible, minimal additional training for customer use cases, and the way the Loras probably work it’s not terrible to pull a few extras. Anyway I just speculate —- no idea what they’re actually doing on the backend.
- liuliu 2y ago
  That's where it is confusing to me. They mentioned that for LoRA fine-tuning, the router weights are frozen, so you don't update the routing when training different concept. But how that expert router is trained? Could be a pretraining with some aux loss to encourage diversity.
badriprof 2y ago
I am old enough to remember the use of yellow pages to find teh right expert

sage92 2y ago

They mention "tuning millions of expert adapters", not 100k

deleted 2y ago

[deleted]

magicalhippo 2y ago

Bit late to the party and I'm not into the AI scene, but from glossing over the three key papers they cite to describe their model, my take is as follows. The idea from LoRA[1] is to take pre-trained, dense weights W_0 for a model and adapt them to new training data by using the weights W = W_0 + BA for inference. The key is that the matrices A and B have a very low rank[2] compared to W_0. Training is then done by treating W_0 as constant, and only updating the A and B matrices. The idea from MoE[3] is to pick just a few "experts" from a large number using Softmax, and insert them between other layers in a neural network. The "experts" can be just some simple matrices or neural networks in their own right. The Lamini model seems to combine these ideas, where they use several "experts" layered between the BA matrices. However instead of just a simple Softmax to select the "experts" they use (chunked?) cross-attention[4], and a much larger number of "experts" compared to the MoE paper. From what I gather the cross-attention allows the expert selection to react to the context, unlike the more naive plain Softmax gating approach. Similarly to LoRA, they train the Lamini model by keeping the pre-trained LLM weights constant. From what I can gather they do a bit of training on the cross-attention layer but then freeze that too, to avoid the cross-attention layer favoring the same "experts" all the time. The idea then is to train the "experts" and the LoRA layer on facts until the combined model (W above) gets each fact correct (zero loss). Thus when the model "sees" a keyword in a sentence, a set of "experts" will steer/adjust the output of the pre-trained LLM to output the correct fact. Or at least that's how I imagine it works. What's less clear to me is what exactly the "experts" are and how they are combined. Since they're used to adjust the weights of the combined model, I it makes the most sense to my un-trained eye that they're simple matrices as mentioned in the MoE paper. Given they're layered between the low-rank portions of the LoRA section, they're necessarily relatively small matrices, so having millions of these "expert" matrices doesn't add too many parameters overall. And while they're portrayed as stacked in the Lamini paper, suggesting matrix multiplication, matrices are generally not commutative[5]. So to the un-trained eye it seems likely the "experts" are simply added like in the MoE paper. But yeah, I'm very much not an expert and the paper was more like a conference poster at best, lacking a lot of detail, so this might be all gibberish and I'd appreciate being corrected. [1]: https://arxiv.org/abs/2106.09685 https://arxiv.org/abs/2106.09685 [2]: https://en.wikipedia.org/wiki/Rank_factorization https://en.wikipedia.org/wiki/Rank_factorization [3]: https://arxiv.org/abs/1701.06538 https://arxiv.org/abs/1701.06538 [4]: https://arxiv.org/abs/2112.04426 https://arxiv.org/abs/2112.04426 [5]: https://en.wikipedia.org/wiki/Commuting_matrices https://en.wikipedia.org/wiki/Commuting_matrices