Interesting counterpoint: a NN is not just activation. It's also learning, and multilayer stacks learn differently depending on depth even if they are entirely

Interesting counterpoint: a NN is not just activation. It's also learning, and multilayer stacks learn differently depending on depth even if they are entirely linear. This seems to be an artefact due to the optimizer in use (SGD for example). The jury seems to be out still on why this is.. the issue has been known since long, and I recently saw a short paper Yann LeCun posted about exactly this as well, they stacked a bunch of completely linear layers on top of a "normal" DL stack and got differences in the final system, even though you can collapse all the linear layers at any time to a single linear mixing layer.