3 ms·

I think this is the overall idea behind the MoE LLM models right? MoE just expands upon this idea by learning which sets of weights to use

kolinko 2y ago
I’d say that this expands on MoE really - MoE chooses dynamically which groups of weights may be needed, but it’s whole matrixes. Here it’s ingle weights. Also, this works on top of MoE beautifully - most of the development and testing was done on Mixtral and it was getting (anecdotally) even better results - getting down to 15-18% effort before seriously losing quality. I decided to release the Mistral version, but Mixtral was fully operational a few commits back :) Also, the cool thing - because you can load only the top say 70% weights, I was running Mixtral full precision on my MB 96G - there were no bemchmarks for this in other impls because others need to load full model into the memory. The real question is Q8 performance - I didn’t implement it fully so far.
- anentropic 2y ago
  Could you add a git tag for the last commit where Mixtral was working?
  - kolinko 2y ago
    I think it was somewhere around that tag: https://github.com/kolinko/effort/releases/tag/5.0-last-mixtral-working https://github.com/kolinko/effort/releases/tag/5.0-last-mixt... Cannot rerun easily any more, because the underlying model/weight names changed in the meantime. It doesn't help that Mixtral's published .safetensor files seem messed up, and I needed to hack a conversion from pytorch - it added an extra layer of confusion into the project.
huac 2y ago
It also feels similar to mixture of depths (https://arxiv.org/abs/2404.02258 https://arxiv.org/abs/2404.02258). Being able to apply this post-training is pretty cool though, makes it easier to use across a wider range of setups.
- kolinko 2y ago
  Yes! I like that, and I saw that paper last weekend iirc. I think these MoD/MoE and other similar methods are highly compatible, and in a similar style. I was originally afraid that this method wouldn't be compatible with MoE and the other methods, but fortunately, at least for Mixtral, there seems to be an amazing synergy. By the way, other tasks have higher priority now, byt there is an interesting observation about MoE. In MoE you get two experts chosen, and each expert has a different weight attached to it - e.g. expert 1 has 75% weight, and expert 2 has 25% weight. Perhaps this could allow to scale the effort to give 75% effort to one expert, and 25% to the other. There are some issues there due to non-linearity of the layers, but perhaps there is something to it.

kolinko 2y ago

I’d say that this expands on MoE really - MoE chooses dynamically which groups of weights may be needed, but it’s whole matrixes. Here it’s ingle weights. Also, this works on top of MoE beautifully - most of the development and testing was done on Mixtral and it was getting (anecdotally) even better results - getting down to 15-18% effort before seriously losing quality. I decided to release the Mistral version, but Mixtral was fully operational a few commits back :) Also, the cool thing - because you can load only the top say 70% weights, I was running Mixtral full precision on my MB 96G - there were no bemchmarks for this in other impls because others need to load full model into the memory. The real question is Q8 performance - I didn’t implement it fully so far.

anentropic 2y ago
Could you add a git tag for the last commit where Mixtral was working?
- kolinko 2y ago
  I think it was somewhere around that tag: https://github.com/kolinko/effort/releases/tag/5.0-last-mixtral-working https://github.com/kolinko/effort/releases/tag/5.0-last-mixt... Cannot rerun easily any more, because the underlying model/weight names changed in the meantime. It doesn't help that Mixtral's published .safetensor files seem messed up, and I needed to hack a conversion from pytorch - it added an extra layer of confusion into the project.

huac 2y ago

It also feels similar to mixture of depths (https://arxiv.org/abs/2404.02258 https://arxiv.org/abs/2404.02258). Being able to apply this post-training is pretty cool though, makes it easier to use across a wider range of setups.

kolinko 2y ago
Yes! I like that, and I saw that paper last weekend iirc. I think these MoD/MoE and other similar methods are highly compatible, and in a similar style. I was originally afraid that this method wouldn't be compatible with MoE and the other methods, but fortunately, at least for Mixtral, there seems to be an amazing synergy. By the way, other tasks have higher priority now, byt there is an interesting observation about MoE. In MoE you get two experts chosen, and each expert has a different weight attached to it - e.g. expert 1 has 75% weight, and expert 2 has 25% weight. Perhaps this could allow to scale the effort to give 75% effort to one expert, and 25% to the other. There are some issues there due to non-linearity of the layers, but perhaps there is something to it.