7 ms·
An "Expert" is really just an unfortunate name for what amounts to a dense part of a sparse matrix and that's also an oversimplification. It doesn't actually s
by Tade0 2mo ago
An "Expert" is really just an unfortunate name for what amounts to a dense part of a sparse matrix and that's also an oversimplification.
It doesn't actually specialise in anything in particular that one can point to.
For this reason you can really transfer them between models.
- 5555watch 2mo agoIs it still true? I'd assume you should be able to freeze the matrix and unfreeze an expert block, before feeding particularly chosen training data. Or that doesn't work?
- Tade0 2mo agoThat's more or less the idea behind Low-Rank Adaptation, or LoRA. There's also Mixture of LoRA Experts, which instead of slicing up the model and routing through that, routes through different LoRAs. But it all comes with tradeoffs, as you have to train and run the gating network doing the routing, which also comes at a cost.