5 ms·

Yup. I got all that from Leo Breiman, not his earlier work as an "academic probabilist" but his later work in classification and regression trees (CART), rando

by graycat 10y ago

Yup. I got all that from Leo Breiman, not his earlier work as an "academic probabilist" but his later work in classification and regression trees (CART), random forests, etc. Yup, if look carefully, in many common cases can do well on prediction accuracy even when parameter accuracy is not so good. E.g., can even do well on prediction accuracy when parameter accuracy is zilch, e.g., some, but not all, cases of over fitting. You are right about the applied math: Machine learning has a lot that is good and new, however the good is mostly not new and the new, not very good. E.g., I watched a Prof Ng lecture on the gradient descent unconstrained minimization involved in their fitting, and it seemed that either their minimization problem is awfully easy or Prof Ng didn't know much about the applied math of such minimization going way back. So, machine learning stole a lot from some now quite classic applied math, assigned some new names, e.g., learning and another anthropomorphic name they dreamed up for iteration step size, and claimed something new. That's called academic theft, plagiarism, or some such, right? Then there's the hype. So, just from machine learning, the pop culture press is extrapolating to the Terminator movies. Absurd. Then, bigger, there's the implicit assumption, if only from just playing with the words, that the main path to smarter machines is machine learning. Well, if the effort is limited to Breiman's piecewise linear curve fitting, that path is absurdly limited. Much better for making machines smarter, i.e., do more, work better, is just to start with lots of now classic pure/applied math, get out a clean sheet of paper, and derive what need for the problem at hand. As much as I like Breiman, both his early work and his last work, there're oceans more good work in applied math that is nothing like Breiman's last work, e.g., with high irony, his earlier work! Uh, here's another case where it looks like a machine has learned -- stochastic optimal control. In some cases, it can look darned smart. Another place -- game theory, to be more specific, the von Neumann-Morgenstern work. Then the Nash generalization. But such things are far from the current machine learning.

srean 10y ago
As I have mentioned several times to you start with vapnik. That's where modern ml begins. Brieman is great but he got a few things wrong, for example he thought boosting and bagging are fundamentally the same, that is they are just variance minimization methods. In fact boosting us what gives random forest its power. BTW less well known trick, use random rotation not just random selection of features or datapoints. Their's lot more to ml than decision trees. One of its fundamental framework is convergence of empirical processes, which is essentially glivenko cantelli on steroids. Vapnik and Chervonenkis provided the first modern breakthrough result but a lot has happened since then. I would say screw the hype just read what it is. I think your disdain for the name 'machine learning' is getting in your way. I strongly disagree with your allegation of academic dishonesty though. Optimization has been a fundamental part of ml from the very start and with a strong overlap in the community. I am sure nemirovskii will ring a bell. In any case how can ml steal from applied math when it is applied math. The lingo differs here and there: estimating parameters become learning parameters etc. Andrew NGO's course is not for you. Go directly to the books and then conference and journal papers. Focus varies but info theory, stats, signal processing, approximation theory are pretty much the same thing, so are their open problems. Btw you could not be more wrong about stochastic optimal control or repeated games being far from current ml. Oh and another major major tool in ml is geometry of function spaces. You are reading the wrong sources, probably the populist ones. Get to the real stuff given what i know of your taste in math i think you will enjoy it.
- selimthegrim 10y ago
  http://amlbook.com http://amlbook.com https://www.youtube.com/watch?v=mbyG85GZ0PI https://www.youtube.com/watch?v=mbyG85GZ0PI (incidentally for graycat, Yaser Abumostafa is a Muslim Egyptian immigrant from Cairo) He covers the VC dimension in https://www.youtube.com/watch?v=Dc0sr0kdBVI https://www.youtube.com/watch?v=Dc0sr0kdBVI, and leaves the proof to an appendix of the book.
  - graycat 10y ago
    Thanks. I looked a few minutes at two of his lectures. I'll keep the URL and watch his lectures during dinners. Lucky he got out of Egypt before they strung him up! Such good people is what US immigration has for some decades now tried to be about. Maybe we will get back to it. "Muslim"? I don't care if he is Zoroastrian either. Or worships some sun god. I don't care about his religion. I do care if he wants to blow up buildings. Somehow I doubt if he does. Looking at his videos, my first cut, crude guess is that he is looking at modern generalizations of old discriminate analysis. Yup, that can be important. Maybe it could be important for, say, one of my old interests, anomaly detection, say, as a doable alternative to Neyman-Pearson where often in practice we don't have nearly enough data. Maybe his interest is in medical diagnosis which, IIRC, was some of Breiman's interest. But, first cut, it looks like, again, the criterion will be, does the model fit the data well? That is, we have little or nothing to recommend the model except that it fits the data well. But, then, in the case of his lectures, it looks like maybe he is making progress to also knowing that the model will predict well. I'm looking forward to how he does that. In contrast, if that is important, in my work in anomaly detection, discussed here on HN often enough, I found false alarm rate from some derivations in applied probability with no model fitting at all. Okay, I don't care if the cat is black or white as long as it catches mice. From a glance, it looks like he is addressing what is meant by learning -- terrific! Not just throwing words around! Then he seems to be addressing when such learning is feasible, etc. Sounds good; I've wondered some about something like that. But my interest now in what he is doing is a bit limited since the core math in my startup seems to be quite different. Thanks.
    - 10y ago

by graycat 10y ago

srean 10y ago

As I have mentioned several times to you start with vapnik. That's where modern ml begins. Brieman is great but he got a few things wrong, for example he thought boosting and bagging are fundamentally the same, that is they are just variance minimization methods. In fact boosting us what gives random forest its power. BTW less well known trick, use random rotation not just random selection of features or datapoints. Their's lot more to ml than decision trees. One of its fundamental framework is convergence of empirical processes, which is essentially glivenko cantelli on steroids. Vapnik and Chervonenkis provided the first modern breakthrough result but a lot has happened since then. I would say screw the hype just read what it is. I think your disdain for the name 'machine learning' is getting in your way. I strongly disagree with your allegation of academic dishonesty though. Optimization has been a fundamental part of ml from the very start and with a strong overlap in the community. I am sure nemirovskii will ring a bell. In any case how can ml steal from applied math when it is applied math. The lingo differs here and there: estimating parameters become learning parameters etc. Andrew NGO's course is not for you. Go directly to the books and then conference and journal papers. Focus varies but info theory, stats, signal processing, approximation theory are pretty much the same thing, so are their open problems. Btw you could not be more wrong about stochastic optimal control or repeated games being far from current ml. Oh and another major major tool in ml is geometry of function spaces. You are reading the wrong sources, probably the populist ones. Get to the real stuff given what i know of your taste in math i think you will enjoy it.

selimthegrim 10y ago
http://amlbook.com http://amlbook.com https://www.youtube.com/watch?v=mbyG85GZ0PI https://www.youtube.com/watch?v=mbyG85GZ0PI (incidentally for graycat, Yaser Abumostafa is a Muslim Egyptian immigrant from Cairo) He covers the VC dimension in https://www.youtube.com/watch?v=Dc0sr0kdBVI https://www.youtube.com/watch?v=Dc0sr0kdBVI, and leaves the proof to an appendix of the book.
- graycat 10y ago
  Thanks. I looked a few minutes at two of his lectures. I'll keep the URL and watch his lectures during dinners. Lucky he got out of Egypt before they strung him up! Such good people is what US immigration has for some decades now tried to be about. Maybe we will get back to it. "Muslim"? I don't care if he is Zoroastrian either. Or worships some sun god. I don't care about his religion. I do care if he wants to blow up buildings. Somehow I doubt if he does. Looking at his videos, my first cut, crude guess is that he is looking at modern generalizations of old discriminate analysis. Yup, that can be important. Maybe it could be important for, say, one of my old interests, anomaly detection, say, as a doable alternative to Neyman-Pearson where often in practice we don't have nearly enough data. Maybe his interest is in medical diagnosis which, IIRC, was some of Breiman's interest. But, first cut, it looks like, again, the criterion will be, does the model fit the data well? That is, we have little or nothing to recommend the model except that it fits the data well. But, then, in the case of his lectures, it looks like maybe he is making progress to also knowing that the model will predict well. I'm looking forward to how he does that. In contrast, if that is important, in my work in anomaly detection, discussed here on HN often enough, I found false alarm rate from some derivations in applied probability with no model fitting at all. Okay, I don't care if the cat is black or white as long as it catches mice. From a glance, it looks like he is addressing what is meant by learning -- terrific! Not just throwing words around! Then he seems to be addressing when such learning is feasible, etc. Sounds good; I've wondered some about something like that. But my interest now in what he is doing is a bit limited since the core math in my startup seems to be quite different. Thanks.
  - 10y ago