4 ms·
Ah, that would stop certain dimensions from changing, but the output would still be the same size.
by duvenaud 8y ago
Ah, that would stop certain dimensions from changing, but the output would still be the same size.
- eximius 8y agoWhile technically still the same size, I think he's proposing that it's, in a sense, isomorphic to a dimension change if the fix to zero is propogates throughout the remainder of the layers (until the next 'change' that is). Or something like that.
- magicalhippo 8y agoI hadn't entirely fleshed out the idea, but yeah. Take a simple NN with 3 layers: 5 neurons in the input layer, 3 in the hidden and 1 output. Force inputs to neuron 4 and 5 in the hidden layer to be zero, and force inputs to neurons 2-5 to be zero in the output layer (and ignore their output). I'm assuming the transfer function obeys f(0) = 0, if not, fix output to zero as well. My thought was this would be similar to how you enforce boundary conditions when solving partial differential equations by directly setting the value of certain matrix elements before running the solver. Again, may be completely silly.