5 ms·
So the time term in these "neural" differential equations is the depth of the network. I see the connection between resnet and the differential equation solver.
by joe_the_user 8y ago
So the time term in these "neural" differential equations is the depth of the network. I see the connection between resnet and the differential equation solver. Basically, you're saying your final result is the end-point of a curve governed by a differential equation whose initial conditions are the input set. You train the equation's parameters in a fashion akin to how you train a standard neural net.
The thing is, with ordinary neural nets you have two arguments for how the thing works.
One view is you are taking inputs and projecting those into a high dimensional feature space and then drawing a separating hyperplanes between points in two or more categories. If you have a proper shaped hyperplanes, the results will tend to generalize.
Another view is you're doing something like logistic regression. You have a function and you're trying to reduce the distance between it and the "true classification function" that takes points to their proper category. You use gradient descent to minimize the distance between your function and this "real function" that you get through data.
So anyway, I am not sure how these intuitions apply to differential equation version. I'm not sure what other intuitions there are for why these equations should work.
- tnecniv 8y ago> Another view is you're doing something like logistic regression. You have a function and you're trying to reduce the distance between it and the "true classification function" that takes points to their proper category. You use gradient descent to minimize the distance between your function and this "real function" that you get through data. This viewpoint doesn't change when you switch to ODEs. The evolution of the system T seconds into the future is a (piecewise) continuous function of the initial state.
- joe_the_user 8y agoMe: Another view is you're doing something like logistic regression.... tnecniv: This viewpoint doesn't change when you switch to ODEs. The evolution of the system T seconds into the future is a (piecewise) continuous function of the initial state. Sure, it doesn't. But it moves to ground where it seems hard to tell what's happening. Approximations of one function by a sum of set functions with variable parameters is a standard arrow in the mathematical "quiver" - Taylor series, linear regression, etc. A neural net naturally is a complex ensemble of such approximations. You gain tremendous flexibility of representation but lose any idea of the shape of the curve that's involved. With the ODE theory, you have a continuous function that is used instead of the parameters. It is hard for me to conceptualize exactly what's happening beyond that. I mean, neural nets seem to able to approximate just about any complex classification function already. So what I believe we need in recasting/representing/etc neural nets is: 1) Ways to speed training. Ways to update a trained system for small-ish changes in the system logic without needing a full retraining. 2) Ways to explain how a neural net reaches its decisions. Ways to tell a neural net to not use a given reasoning branch if it's using "garbage reasoning". 3) Ways to merge several neural nets that operate on related but not identical problems. Etc. I'm not sure if this model brings us closer to this.