It’s absolutely beneficial when training because the forward pass and back propagation is still only on the neurons that were activated. The Mistral guys speci

It’s absolutely beneficial when training because the forward pass and back propagation is still only on the neurons that were activated. The Mistral guys specifically mention that training speed (due to not needing as much compute) was one of the reasons Mixtral was released so soon after Mistal 7b.