3 ms·

> Another view is you're doing something like logistic regression. You have a function and you're trying to reduce the distance between it and the "true classif

> Another view is you're doing something like logistic regression. You have a function and you're trying to reduce the distance between it and the "true classification function" that takes points to their proper category. You use gradient descent to minimize the distance between your function and this "real function" that you get through data. This viewpoint doesn't change when you switch to ODEs. The evolution of the system T seconds into the future is a (piecewise) continuous function of the initial state.

joe_the_user 8y ago
Me: Another view is you're doing something like logistic regression.... tnecniv: This viewpoint doesn't change when you switch to ODEs. The evolution of the system T seconds into the future is a (piecewise) continuous function of the initial state. Sure, it doesn't. But it moves to ground where it seems hard to tell what's happening. Approximations of one function by a sum of set functions with variable parameters is a standard arrow in the mathematical "quiver" - Taylor series, linear regression, etc. A neural net naturally is a complex ensemble of such approximations. You gain tremendous flexibility of representation but lose any idea of the shape of the curve that's involved. With the ODE theory, you have a continuous function that is used instead of the parameters. It is hard for me to conceptualize exactly what's happening beyond that. I mean, neural nets seem to able to approximate just about any complex classification function already. So what I believe we need in recasting/representing/etc neural nets is: 1) Ways to speed training. Ways to update a trained system for small-ish changes in the system logic without needing a full retraining. 2) Ways to explain how a neural net reaches its decisions. Ways to tell a neural net to not use a given reasoning branch if it's using "garbage reasoning". 3) Ways to merge several neural nets that operate on related but not identical problems. Etc. I'm not sure if this model brings us closer to this.

joe_the_user 8y ago

Me: Another view is you're doing something like logistic regression.... tnecniv: This viewpoint doesn't change when you switch to ODEs. The evolution of the system T seconds into the future is a (piecewise) continuous function of the initial state. Sure, it doesn't. But it moves to ground where it seems hard to tell what's happening. Approximations of one function by a sum of set functions with variable parameters is a standard arrow in the mathematical "quiver" - Taylor series, linear regression, etc. A neural net naturally is a complex ensemble of such approximations. You gain tremendous flexibility of representation but lose any idea of the shape of the curve that's involved. With the ODE theory, you have a continuous function that is used instead of the parameters. It is hard for me to conceptualize exactly what's happening beyond that. I mean, neural nets seem to able to approximate just about any complex classification function already. So what I believe we need in recasting/representing/etc neural nets is: 1) Ways to speed training. Ways to update a trained system for small-ish changes in the system logic without needing a full retraining. 2) Ways to explain how a neural net reaches its decisions. Ways to tell a neural net to not use a given reasoning branch if it's using "garbage reasoning". 3) Ways to merge several neural nets that operate on related but not identical problems. Etc. I'm not sure if this model brings us closer to this.