4 ms·

ML is a souped-up version of "draw a line through these points". We can get really efficient at drawing lines through points, but it's not like we'll suddenly r

by xamuel 8y ago

ML is a souped-up version of "draw a line through these points". We can get really efficient at drawing lines through points, but it's not like we'll suddenly realize some deeper fundamental theory about it!

whatshisface 8y ago
The reason you don't expect to see a deep fundamental theory of drawing a line through a few points is because you can always do it. ML doesn't always work, and sometimes it is harder to get working than other times. What's going on?
- xamuel 8y ago
  You can always draw a line through the points, but it isn't always a good approximation. If the points are inherently bunched around a line, then a line through them will approximate them well. If they're a big random cloud, then the line won't. It's the exact same way in ML, except the points are in n-dimensional space and "line" is replaced by "higher-dimensional curve or manifold". Sometimes (e.g. in image processing), the n-dimensional points are inherently bunched around a curve or manifold of the form you're using, and then ML works great. There's nothing deeper going on!
wish5031 8y ago
But there actually is a huge amount of theory behind that problem. You can exactly derive the method that finds the best line. You can get error bounds on each of your coefficients and confidence intervals for them. You can alter the strength of your assumptions (e.g. about distribution of errors, homoskedacity, and so on) and see how it affects your model. You can add L1 or L2 regularization, both of which also have solid theoretical grounding. And so on. All of these things help make your model more robust and give you greater confidence in it, which will be important if we want to put ML in, say, healthcare or defense. But you don’t get as much of this theory with more complex ML models, and certainly not with neural nets. Good luck trying to get a confidence interval for the optimal value of a weight in your net, much less interpreting it.

by xamuel 8y ago

whatshisface 8y ago

The reason you don't expect to see a deep fundamental theory of drawing a line through a few points is because you can always do it. ML doesn't always work, and sometimes it is harder to get working than other times. What's going on?

xamuel 8y ago
You can always draw a line through the points, but it isn't always a good approximation. If the points are inherently bunched around a line, then a line through them will approximate them well. If they're a big random cloud, then the line won't. It's the exact same way in ML, except the points are in n-dimensional space and "line" is replaced by "higher-dimensional curve or manifold". Sometimes (e.g. in image processing), the n-dimensional points are inherently bunched around a curve or manifold of the form you're using, and then ML works great. There's nothing deeper going on!

wish5031 8y ago

But there actually is a huge amount of theory behind that problem. You can exactly derive the method that finds the best line. You can get error bounds on each of your coefficients and confidence intervals for them. You can alter the strength of your assumptions (e.g. about distribution of errors, homoskedacity, and so on) and see how it affects your model. You can add L1 or L2 regularization, both of which also have solid theoretical grounding. And so on. All of these things help make your model more robust and give you greater confidence in it, which will be important if we want to put ML in, say, healthcare or defense. But you don’t get as much of this theory with more complex ML models, and certainly not with neural nets. Good luck trying to get a confidence interval for the optimal value of a weight in your net, much less interpreting it.