Yeah, dropout is conspicuously missing. For those who haven't heard: Dropout zeros out random activations, creating a random subnetwork to make a prediction. Ea

Yeah, dropout is conspicuously missing. For those who haven't heard: Dropout zeros out random activations, creating a random subnetwork to make a prediction. Each training step updates a different random subnetwork. (...Keeping in mind that two random subnetworks can share lots of weights...) Then at inference time, you use the full network, which now acts as an ensemble of all of the subnetworks. You could view it as 'thousand brains' with lots of weight sharing for efficiency+regularization.