Makes me wonder what the SOTA is for open source efforts along these lines. I have heard about "mixture of experts" as being a potentially important advance, a

Makes me wonder what the SOTA is for open source efforts along these lines. I have heard about "mixture of experts" as being a potentially important advance, and also of course about multimodality. So I found this: https://github.com/YeonwooSung/LIMoE-pytorch https://github.com/YeonwooSung/LIMoE-pytorch