5 ms·
I do think it's misleading to compare the method to a general feed-forward network though, for two reasons. First, to preserve the analogy between eq. 1 and 2,
by soVeryTired 8y ago
I do think it's misleading to compare the method to a general feed-forward network though, for two reasons.
First, to preserve the analogy between eq. 1 and 2, the thetas in equation two should have their own dynamics, which should be learned.
Second, even if Equation 1 doesn't allow it, in a general feed-forward network it's possible for the state to change dimension between layers. I don't see how that could happen with the continuous model.
Neat paper, but it'd be nice if they had tied the analogy more explicitly to RNNs in the introduction.
- duvenaud 8y agoThe comparison we make is to residual networks, which I think is valid. First, we do parameterize a theta that changes with time, using a hypernet. But this is equivalent to the way sampo wrote the equations above - you can just feed time as another input to the dynamics network to get dynamics that change with time. Second, I agree that general feedforward nets allow dimension changes, but resnets don't. This model is a drop-in replacement for resnets, but not for any feedforward net. If we gave the wrong impression somewhere, please let us know. We didn't make the analogy with RNNs, because I don't think it fits - standard input-output RNNs have to take in part of the data with every time step, while here the data only appears at the input (depth 0) layer of the network.
- joe_the_user 8y agoHasn't there been similar work to this in the past? I don't see a "related work" section in your paper.
- duvenaud 8y agoSection 7 is titled "Related Work".
- joe_the_user 8y agoThanks, Apologies for not looking harder
- soVeryTired 8y agoYou're absolutely right - sorry, somehow I managed to miss the explicit time parameter in your equation two, and didn't read carefully enough to see that you were restricting the discussion to resnets and normalising flows. You might be able to make a better connection to RNNs by having the input data as a 'forcing' function in your ODE. But you probably need some regularity conditions on the input data to make sure the result is nicely behaved.
- im3w1l 8y ago> standard input-output RNNs have to take in part of the data with every time step Well they can but I don't see they why they have to. And couldn't your network also take input and give output for all times? > First, we do parameterize a theta that changes with time, using a hypernet. Ah I see. Did you end up using it in the final model? I don't see that in the mnist example, but I could be missing it as I only skimmed the code.
- magicalhippo 8y agoNow it's been ages since I dabbled with neural nets so this might be completely silly, but can't a change in dimension be thought of as forcing the weights to/from certain nodes to be zero?
- duvenaud 8y agoAh, that would stop certain dimensions from changing, but the output would still be the same size.
- eximius 8y agoWhile technically still the same size, I think he's proposing that it's, in a sense, isomorphic to a dimension change if the fix to zero is propogates throughout the remainder of the layers (until the next 'change' that is). Or something like that.
- magicalhippo 8y agoI hadn't entirely fleshed out the idea, but yeah. Take a simple NN with 3 layers: 5 neurons in the input layer, 3 in the hidden and 1 output. Force inputs to neuron 4 and 5 in the hidden layer to be zero, and force inputs to neurons 2-5 to be zero in the output layer (and ignore their output). I'm assuming the transfer function obeys f(0) = 0, if not, fix output to zero as well. My thought was this would be similar to how you enforce boundary conditions when solving partial differential equations by directly setting the value of certain matrix elements before running the solver. Again, may be completely silly.