3 ms·

> Much of machine learning is done empirically. Yes, it does seem so. > Mathematical guarantees are interesting, but really who cares? We should all care. T

by graycat 13y ago

> Much of machine learning is done empirically. Yes, it does seem so. > Mathematical guarantees are interesting, but really who cares? We should all care. The guaranteed help us know what the heck we are doing and what the heck we've got when we are done. E.g., in statistics we like estimators that are unbiased and minimum variance -- that's 'goodness' and good to know and potentially valuable in practice. Estimators that are biased and with variance larger than necessary are not good. We like regression in part because it is 'least squares' which means it is a projection and, thus, in mean-square sense, the most accurate approximation we can get. We like statistics with loss and risk functions because they tell us what to do at least cost. Etc.

srean 13y ago
Unbiasedness is oversold. It makes sense when you also average the estimator, otherwise what matters is the expected error or expected loss (probabilities thereof). Let the error be in the bias part or the variance part, what do I care. Furthermore not all losses decompose into bias and variance either. I also think the fascination with least squares led us astray. Its a bad loss function for classification. Bad for neural networks that use the sigmoid function. Bad because it can create exponentially many local minima, on the other hand with sigmiod function if you used KL divergence instead of square loss you will not have these problems. Part of it is the old statistical vestige of analyzing everything using tools of asymptotic Normality. But I agree with your general point, (and upvoted your parent comment) one should care about what one is really getting after the algorithm has been applied, and phrase that in language of mathematics. If the community does not do this a) the snake-oil salesmen will come out of the woodworks and b) machine learning will suffer the same fate as AI winter c) at the same time one should not discourage open ended practice and experimentation (that is how most new things are discovered, analysis comes later, mostly)
- graycat 13y ago
  On your last paragraph, we agree. Moreover, your a), b) say some of what I think better than I said it. Otherwise you have a good critique of traditional statistics. On traditional statistics, I voted with my research and did work that was distribution-free, also multidimensional, unusual for distribution-free. I did wonder if there could be another way, and what you wrote about Vapnik suggests that there can be. This thread now is much deeper into the analysis of machine learning algorithms than I noticed in what I saw of Stanford professor Ng's course and several books and papers in PDF I downloaded and scanned. I would guess the level of Vapnik's work would be that of a graduate seminar and not a ugrad computer science course and that the mathematical prerequisites would filter out nearly everyone with a ugrad CS degree and nearly all their professors. Why? From some of my experience getting a paper of mine reviewed, measure preserving totally blew away several editors in chief of famous CS journals and chaired professors of CS at famous research universities; one such person wrote me "Neither I nor anyone on my board of editors has the mathematical prerequisites to review your paper.", and a second such person wrote me a similar remark. The math prerequisites were not so advanced, say, just at the level a graduate course in probability via, say, Loeve or Neveu and fast and easy reading for, say, E. Cinlar, P. Billingsley, L. Breiman, J. Doob, A. Karr, E. Dynkin, etc. But the paper was 'CS' because it was on a CS problem. For AI, I cooked up the paper to improve on some of our work with 'expert systems' for the problem. I have to suspect that work in machine learning that needs some of Loeve or functional analysis from, say, Rudin will not be popular in CS departments. This thread has become a good start on a grad seminar. Just now I can't follow up on the leads here; instead I have to finish off the code for two Web pages and get my new Web site live.
  - srean 13y ago
    Machine learning is really one part probability theory, one part functional analysis one part optimization and another part algorithms. I think with your background you will enjoy it. The particular two papers by Vapnik and Chervonenkis are very readable and does not use anything more technical than Chernoff bounds, its more about how they are used rather than what are used. CS students typically do not have probability in their curriculum, at least no measure theory, so you will see a lot of Physicist turned machine learners in this field. You might have something interesting going on with your own research finding, if you have not published it I would encourage you to do so. A possible venue is JMLR (journal of machine learning research, there editors are not scared of a little measure theory). BTW a surprising fact is that Vapnik's bounds are dimensionality independent, provided you are working with what are known as function classes with finite VC (Vapnki-Chervonenkis) dimension. Dimensionality is a big deal in m/c learning because the data they are interested in is typically quite high dimensional.
    - graycat 13y ago
      > You might have something interesting going on with your own research finding, if you have not published it I would encourage you to do so. 'Information Sciences', 1999, "... Anomaly Detector". So look for 'zero day' problems in server farms and networks. Get a multidimensional, distribution-free hypothesis test where know false alarm rate in advance and can adjust it exactly in small steps over a wide range. What is novel is the 'applied probability' that permits the results with false alarm rate. The assumptions about the input data are realistic and, thus, do not permit doing as well as the Neyman-Pearson result, but, still, intuitively, the detection rate should be relatively high. There is a weaker sense, not in the paper, where the detection rate is as high as possible. I.e., essentially, for the selected false alarm rate, the 'critical region' where null hypothesis (a healthy system) is accepted has least possible Lebesgue measure. It appears that the input data can have dimensions 10, 100, 1000, etc. I have an algorithm that makes the computations fast, but it's not in the paper. The computing needed was a bit much when I published the paper, but that computing is quite reasonable now. For a system where really care a lot about reliability, performance, security, etc., it could be worthwhile to deploy what I cooked up. My current project is quite different but does need a server farm, and I might deploy my detector there. Then with that demo, if people like my detector, I might let people buy it.
Houshalter 13y ago
Perhaps, but IMHO the obsession with "provably optimal" is bad and will never lead to anything useful. The real world is messy and complicated. Search spaces are far too large to ever completely explore. We need heuristics to do things in a reasonable amount of time. Heuristics that exploit the structure of the problem we are trying to solve. And those are going to be difficult if not impossible to formally verify.

by graycat 13y ago

srean 13y ago

Unbiasedness is oversold. It makes sense when you also average the estimator, otherwise what matters is the expected error or expected loss (probabilities thereof). Let the error be in the bias part or the variance part, what do I care. Furthermore not all losses decompose into bias and variance either. I also think the fascination with least squares led us astray. Its a bad loss function for classification. Bad for neural networks that use the sigmoid function. Bad because it can create exponentially many local minima, on the other hand with sigmiod function if you used KL divergence instead of square loss you will not have these problems. Part of it is the old statistical vestige of analyzing everything using tools of asymptotic Normality. But I agree with your general point, (and upvoted your parent comment) one should care about what one is really getting after the algorithm has been applied, and phrase that in language of mathematics. If the community does not do this a) the snake-oil salesmen will come out of the woodworks and b) machine learning will suffer the same fate as AI winter c) at the same time one should not discourage open ended practice and experimentation (that is how most new things are discovered, analysis comes later, mostly)

graycat 13y ago
On your last paragraph, we agree. Moreover, your a), b) say some of what I think better than I said it. Otherwise you have a good critique of traditional statistics. On traditional statistics, I voted with my research and did work that was distribution-free, also multidimensional, unusual for distribution-free. I did wonder if there could be another way, and what you wrote about Vapnik suggests that there can be. This thread now is much deeper into the analysis of machine learning algorithms than I noticed in what I saw of Stanford professor Ng's course and several books and papers in PDF I downloaded and scanned. I would guess the level of Vapnik's work would be that of a graduate seminar and not a ugrad computer science course and that the mathematical prerequisites would filter out nearly everyone with a ugrad CS degree and nearly all their professors. Why? From some of my experience getting a paper of mine reviewed, measure preserving totally blew away several editors in chief of famous CS journals and chaired professors of CS at famous research universities; one such person wrote me "Neither I nor anyone on my board of editors has the mathematical prerequisites to review your paper.", and a second such person wrote me a similar remark. The math prerequisites were not so advanced, say, just at the level a graduate course in probability via, say, Loeve or Neveu and fast and easy reading for, say, E. Cinlar, P. Billingsley, L. Breiman, J. Doob, A. Karr, E. Dynkin, etc. But the paper was 'CS' because it was on a CS problem. For AI, I cooked up the paper to improve on some of our work with 'expert systems' for the problem. I have to suspect that work in machine learning that needs some of Loeve or functional analysis from, say, Rudin will not be popular in CS departments. This thread has become a good start on a grad seminar. Just now I can't follow up on the leads here; instead I have to finish off the code for two Web pages and get my new Web site live.
- srean 13y ago
  Machine learning is really one part probability theory, one part functional analysis one part optimization and another part algorithms. I think with your background you will enjoy it. The particular two papers by Vapnik and Chervonenkis are very readable and does not use anything more technical than Chernoff bounds, its more about how they are used rather than what are used. CS students typically do not have probability in their curriculum, at least no measure theory, so you will see a lot of Physicist turned machine learners in this field. You might have something interesting going on with your own research finding, if you have not published it I would encourage you to do so. A possible venue is JMLR (journal of machine learning research, there editors are not scared of a little measure theory). BTW a surprising fact is that Vapnik's bounds are dimensionality independent, provided you are working with what are known as function classes with finite VC (Vapnki-Chervonenkis) dimension. Dimensionality is a big deal in m/c learning because the data they are interested in is typically quite high dimensional.
  - graycat 13y ago
    > You might have something interesting going on with your own research finding, if you have not published it I would encourage you to do so. 'Information Sciences', 1999, "... Anomaly Detector". So look for 'zero day' problems in server farms and networks. Get a multidimensional, distribution-free hypothesis test where know false alarm rate in advance and can adjust it exactly in small steps over a wide range. What is novel is the 'applied probability' that permits the results with false alarm rate. The assumptions about the input data are realistic and, thus, do not permit doing as well as the Neyman-Pearson result, but, still, intuitively, the detection rate should be relatively high. There is a weaker sense, not in the paper, where the detection rate is as high as possible. I.e., essentially, for the selected false alarm rate, the 'critical region' where null hypothesis (a healthy system) is accepted has least possible Lebesgue measure. It appears that the input data can have dimensions 10, 100, 1000, etc. I have an algorithm that makes the computations fast, but it's not in the paper. The computing needed was a bit much when I published the paper, but that computing is quite reasonable now. For a system where really care a lot about reliability, performance, security, etc., it could be worthwhile to deploy what I cooked up. My current project is quite different but does need a server farm, and I might deploy my detector there. Then with that demo, if people like my detector, I might let people buy it.

Houshalter 13y ago

Perhaps, but IMHO the obsession with "provably optimal" is bad and will never lead to anything useful. The real world is messy and complicated. Search spaces are far too large to ever completely explore. We need heuristics to do things in a reasonable amount of time. Heuristics that exploit the structure of the problem we are trying to solve. And those are going to be difficult if not impossible to formally verify.