Do I understand correctly that they are showing, "if we increase the size of these tensors along these directions, by padding them with zeros, it does not chang

Do I understand correctly that they are showing, "if we increase the size of these tensors along these directions, by padding them with zeros, it does not change the inference behavior of the network."? If so, I'm.. kind of surprised that this wasn't found earlier? Ah, well, they also show that they can also insert a new layer in the middle, but, because the layers output changes to be added on to the vector that was received, it does not seem terribly surprising to me that they can initialize a layer such that it produces 0 as the vector it should add onto it. Ah, wait, no, they also show that other parts of tensors that are added when increasing the size of some parameters, can be initialized arbitrarily, provided that the appropriate other parts are 0 (or, either 0 or identity matrices? not sure.) And, also some of the tensors have to be re-scaled. Ok. edit: Though... seeing as a new layer can be inserted at any point, this does make me wonder about, could we do some pre-training of a network, and then freeze those weights, add in a layer and train that layer on some fine-tuning task, and then, separately, do the same thing with a number of other fine-tuning tasks (each one with the extra layer being trained without the others being present), and then like, take the network where each of those extra layers are present at the same position in parallel (so, they get the same input, and each of their outputs are added to the sum being created, before going on to the next step. Maybe with each of them being weighted.) This seems like it could allow for some kind of like, mixture of experts thing? idk what I'm talking about though