> MoD transformers demonstrate the value of routing among different types of computations. In this work the types were either the conventional transformer block

> MoD transformers demonstrate the value of routing among different types of computations. In this work the types were either the conventional transformer block, or a null computation (functionally equivalent to multiplying by zero). However, one can imagine extending this idea further by routing between even more types of computation. For example, perhaps some tokens are routed to "memory lookup" functions, and others are routed to "tool use" functions. This is one of the most interesting parts of the paper for me. It seems that this could be a much more efficient approach for tool use and math calculation.