4 ms·

> hanging over the whole subject is a suspicion that, really, as nice as the general theories are, mostly the applications are just a few, standard differentia

> hanging over the whole subject is a suspicion that, really, as nice as the general theories are, mostly the applications are just a few, standard differential equations I see something similar when it comes to machine learning. If you start to really dig into the underpinnings of the topic, you find fairly complex things like partial derivatives (for gradient descent optimization, for example), but you don't really have to understand it much to take it, apply it, and verify that the results make sense. On the other hand, I've been around long enough to have learned the hard way that applying something you don't really, fully, from-the-ground-up comprehend can bite you in surprising ways.

graycat 8y ago
I'd suggest learning about partial derivatives and in particular, the gradient. With an appropriate book, one evening should be enough. Intuitively the negative of the gradient is the direction to ski fastest downhill, and that's why it is heavily involved in optimization problems. Then there are close connections with convexity -- intuitively the inside of most kitchen cereal bowls is convex. The gradient is crucial in uses of Lagrange multipliers in the non-linear cases; in non-linear optimization with constraints, the gradient is central to the Kuhh-Tucker-Karush necessary conditions for optimality. If are in a deep, long, narrow river valley, want to get to the bottom downstream of the river, and ski downhill using the gradient, then will keep crossing the river over and over, traveling many feet across the river for each foot going downstream. So can approximate the valley with an ellipse and ski in much better direction along the long axis of the ellipse. People figured this out long ago -- it's called conjugate gradients. If the river wanders, then that's still more difficult. In the best fitting in ML, may be in such river valleys, and some notes on ML recognize this and warn about using just the gradient. A lot more with gradient is known and at times useful -- Newton iteration, quasi-Newton, etc.
- srean 8y ago
  Vigorously seconded. There is a lot of applications of convex analysis, convex duality, KKT conditions and game theory in ML if one looks at it right. In fact the thats at the very foundation of techniques such as support vector machines (large margin separators in a Hilbert space), regret minimization algorithms, etc etc. Of course one can choose to ignore all that and only focus on stochastic gradient descent. That will carry one for some non-trivial distance.
graycat 8y ago
> If you start to really dig into the underpinnings of the topic, Let's cover the most important part of partial derivatives, the geometric intuition. Imagine the Smoky Mountains of east Tennessee, that is, smooth, rolling hills. Now to represent this landscape in math, let R be the set of real numbers, R^2 pairs of real numbers, that is, the coordinates of points in the plane with orthogonal axes X and Y, and let f: R^2 --> R, that is, f is a function of two variables, say, x and y, that is, the pair (x,y) in R^2, and the value f(x,y) is the height of the mountains above point (x,y), that is, a plane under the mountains. Then at a point (x, y) the partial derivative of f(x,y) with respect to x is just the slope as in ordinary derivative of the mountain at point (x,y) in the direction of changing x. So, if the X axis runs east and west, the partial derivative of f(x,y) with respect to x is the slope of the mountain at (x,y) in the east-west direction. So the partial derivative is just like the derivative of a function of one variable, that is, a slope, except is for just one variable, say, x, with the other variable(s) y held constant. So, if f(x,y) = 3xy + 2x - y then the partial derivative of f(x,y) with respect to x is just D_x f(x,y) = 3y + 2 and the partial derivative of f(x,y) with respect to y is D_y f(x,y) = 3x - 1 These partial derivatives are important in vector analysis and, thus, Maxwell's equations, electro-magnetism, fluid flow, optimization, etc.

graycat 8y ago

I'd suggest learning about partial derivatives and in particular, the gradient. With an appropriate book, one evening should be enough. Intuitively the negative of the gradient is the direction to ski fastest downhill, and that's why it is heavily involved in optimization problems. Then there are close connections with convexity -- intuitively the inside of most kitchen cereal bowls is convex. The gradient is crucial in uses of Lagrange multipliers in the non-linear cases; in non-linear optimization with constraints, the gradient is central to the Kuhh-Tucker-Karush necessary conditions for optimality. If are in a deep, long, narrow river valley, want to get to the bottom downstream of the river, and ski downhill using the gradient, then will keep crossing the river over and over, traveling many feet across the river for each foot going downstream. So can approximate the valley with an ellipse and ski in much better direction along the long axis of the ellipse. People figured this out long ago -- it's called conjugate gradients. If the river wanders, then that's still more difficult. In the best fitting in ML, may be in such river valleys, and some notes on ML recognize this and warn about using just the gradient. A lot more with gradient is known and at times useful -- Newton iteration, quasi-Newton, etc.

srean 8y ago
Vigorously seconded. There is a lot of applications of convex analysis, convex duality, KKT conditions and game theory in ML if one looks at it right. In fact the thats at the very foundation of techniques such as support vector machines (large margin separators in a Hilbert space), regret minimization algorithms, etc etc. Of course one can choose to ignore all that and only focus on stochastic gradient descent. That will carry one for some non-trivial distance.

graycat 8y ago

> If you start to really dig into the underpinnings of the topic, Let's cover the most important part of partial derivatives, the geometric intuition. Imagine the Smoky Mountains of east Tennessee, that is, smooth, rolling hills. Now to represent this landscape in math, let R be the set of real numbers, R^2 pairs of real numbers, that is, the coordinates of points in the plane with orthogonal axes X and Y, and let f: R^2 --> R, that is, f is a function of two variables, say, x and y, that is, the pair (x,y) in R^2, and the value f(x,y) is the height of the mountains above point (x,y), that is, a plane under the mountains. Then at a point (x, y) the partial derivative of f(x,y) with respect to x is just the slope as in ordinary derivative of the mountain at point (x,y) in the direction of changing x. So, if the X axis runs east and west, the partial derivative of f(x,y) with respect to x is the slope of the mountain at (x,y) in the east-west direction. So the partial derivative is just like the derivative of a function of one variable, that is, a slope, except is for just one variable, say, x, with the other variable(s) y held constant. So, if f(x,y) = 3xy + 2x - y then the partial derivative of f(x,y) with respect to x is just D_x f(x,y) = 3y + 2 and the partial derivative of f(x,y) with respect to y is D_y f(x,y) = 3x - 1 These partial derivatives are important in vector analysis and, thus, Maxwell's equations, electro-magnetism, fluid flow, optimization, etc.