3 ms·

The full layer is skipped. I.e. replaced by identity. Sth like g(x) = switch(prob, x, f(x)). Deep Residual Networks are similar but different. There, you add w

• radarsat1 11y ago
  Yes, but my question was more, when the layer is "skipped", what happens to the input for the next layer? But clearly, it is designed such that the identity function still provides somehow useful information to the next layer. (i.e doesn't significantly transform its domain and range) I was wondering how this could work. It's just that, intuitively, I would think that the next layer is being trained on a specific transformation performed by the skipped layer, so I still don't fully understand how replacing a whole layer with the identity function doesn't completely mess up the training of all subsequent layers. But maybe the secret is that it only lasts for a small number of iterations, and perhaps this short-lived deviation actually helps inject some minima-escaping trajectory. (I have read that injecting random noise can have similar effects. Is this just a different kind of random noise?)