Careless application of Gumbel Softmax would let your model to select same input for both inputs of a gate. Also, note that selecting a wire has all same probl

Careless application of Gumbel Softmax would let your model to select same input for both inputs of a gate. Also, note that selecting a wire has all same problems as selecting an operation, such as vanishing gradient, but no apparent mitigation like residual initialization. And for wire selection gradient vanishes quicker, because there are more wires (tens and hundredths) than operations (16).