As mentioned that would be more like mixture-of-experts, where you instead effectively just multiply the rows with some of the columns in your weight matrices.

As mentioned that would be more like mixture-of-experts, where you instead effectively just multiply the rows with some of the columns in your weight matrices. That said, I wrote my post after bed time and I'm now pretty sure I wasn't thinking straight.