5 ms·

Hey sorry but this > In addition, too much of ML is .... short on math guarantees. is utterly false. As I keep telling you need to upgrade :) In fact I will

by srean 9y ago

Hey sorry but this > In addition, too much of ML is .... short on math guarantees. is utterly false. As I keep telling you need to upgrade :) In fact I will level the same exact criticism on standard stats. NP lemma is fantastic math but of little practical importance because you rarely ever know the densities. Most of the time 'test of hypothesis' is a bad joke catering to a contrived scenario. Real problems are rarely about 'is it H0 or H1'. Estimation is typically more useful. Stats have long suffered from their fetish for parametric models, asymptotic normality, asymptotic guarantees and what i feel their greatest deficiency: the focus on parameter estimation rather than guarantees over prediction. Parameters and parametric models are convenient fiction, nobodody has seen them, will ever see them. What's all the fuss about estimating a fictional object to great accuracy under strong assumptions in infinite time about. Made sense a century ago though. If their purported utility is in aiding prediction then why not go after the real deal, that is, non-asymptotic guarantees on prediction in a distribution free / nonparametric setting. That is what ML does. Please dont characterize ML by bad examples and work in progress stuff. I do know nonparametric stats exist but thats not the face of stats that people get to see, but even in those cases the focus is usually on accuracy of estimating a functional. I was deliberately harsh on stats in my comment to offset some of the wrong things you mentioned about stats, but the relation between ml and stats are really not adversarial. I see ML as a course correction for stats: focus on prediction guarantees and adding some muscle in two areas (i) algorithms for scaling (ii) optimization. In anycase i know where both the tribe hide their dirty laundry :) That said I strongly agree that Ml is indeed applied math a mix of probability, optimization, functional analysis and algorithms/Datastructures

graycat 9y ago
I agree with a lot of that. But hypothesis testing remains important because in some contexts with too little data that's about the best can do. For parametric tests, there are some cases when that's okay. The asymptotic stuff is mostly okay: Can't figure out precisely what the darned thing will do for case n so let n go to infinity and maybe can see what happens. Then say, "for large n, this is about what happens" -- crude but better than nothing. The central limit theorem and more says that the Gaussian is really important. Okay. Done. We know that. But, right, for too long too much of statistics made nearly a religion out of the Gaussian. Bummer. Today happened to see the discussion panel at Stanford at https://www.youtube.com/watch?v=hxXIJnjC_HI https://www.youtube.com/watch?v=hxXIJnjC_HI It's fun stuff, with some big names. They are laughing at ML: ML discovered statistics like Columbus discovered America. Columbus didn't discover America since when he landed millions of people were already living there. And when CS AL/ML discovered statistics, there was already a huge field there. Moreover, the situation is reversed: Columbus brought better ship, etc. technology to the Americas than the Americas had, but CS AL/ML are bringing nearly all highly inferior technology to statistics, pure/applied math, etc. CS AI/ML is nearly all new labels for adulterated old wine. So, take some data on two variables, plot on an X-Y graph, fit a straight line, get the coefficients of the line as in first year algebra, and, presto, bingo, "The AI revolution is here!!!!", CS had had a machine "learn" the coefficients and they started with "training" data, that is trained the machine!!!!!! Where can I get one of those airline seat barf bags? Upchuck time! Uh, I have a pretty good professional library; much of my house is lined with -bookshelves. The main topics are pure/applied math. Some of the books relevant to statistics include (with TeX markup) Alexander M.\ Mood, Franklin A.\ Graybill, and Duane C.\ Boas, {\it Introduction to the Theory of Statistics, Third Edition,\/} McGraw-Hill, New York, 1974.\ \ N.\ R.\ Draper and H.\ Smith, {\it Applied Regression Analysis,\/} John Wiley and Sons, New York, 1968.\ \ C.\ Radhakrishna Rao, {\it Linear Statistical Inference and Its Applications:\ \ Second Edition,\/} ISBN 0-471-70823-2, John Wiley and Sons, New York, 1967.\ \ Henry Scheff\'e, {\it Analysis of Variance,\/} John Wiley and Sons, New York, 1967.\ \ Yvonne M.\ M.\ Bishop, Stephen E.\ Fienberg, Paul W.\ Holland, {\it Discrete Multivariate Analysis:\ \ Theory and Practice,\/} ISBN 0-262-52040-0, MIT Press, Cambridge, Massachusetts, 1979.\ \ Stephen E.\ Fienberg, {\it The Analysis of Cross-Classified Data,\/} ISBN 0-262-06063-9, MIT Press, Cambridge, Massachusetts, 1979.\ \ Leo Breiman, Jerome H.\ Friedman, Richard A.\ Olshen, Charles J.\ Stone, {\it Classification and Regression Trees,\/} ISBN 0-534-98054-6, Wadsworth \& Brooks/Cole, Pacific Grove, California, 1984.\ \ R.\ B.\ Blackman and J.\ W.\ Tukey, {\it The Measurement of Power Spectra:\ \ From the Point of View of Communications Engineering,\/} Dover, New York, 1959.\ \ William W.\ Cooley and Paul R.\ Lohnes, {\it Multivariate Data Analysis,\/} John Wiley and Sons, New York, 1971.\ \ Maurice M.\ Tatsuoka, {\it Multivariate Analysis: Techniques for Educational and Psychological Research,\/} John Wiley and Sons, 1971.\ \ E.\ L.\ Lehmann, {\it Testing Statistical Hypotheses,\/} John Wiley, New York, 1959.\ \ E.\ L.\ Lehmann, {\it Nonparametrics: Statistical Methods Based on Ranks,\/} ISBN 0-8162-4994-6, Holden-Day, San Francisco, 1975.\ \ Jaroslav H\'ajek and Zbyn\v ek \v Sid\'ak, {\it Theory of Rank Tests,\/} Academia, Prague, 1967.\ \ Sidney Siegel, {\it Nonparametric Statistics for the Behavioral Sciences,\/} McGraw-Hill, New York, 1956.\ \ Donald F.\ Morrison, {\it Multivariate Statistical Methods: Second Edition,\/} ISBN 0-07-043186-8, McGraw-Hill, New York, 1976.\ \ Harry H.\ Harman, {\it Modern Factor Analysis: Second Edition, Revised,\/} The University of Chicago Press, Chicago, 1967.\ \ George E.\ P.\ Box and Gwilym M.\ Jenkins, {\it Time Series Analysis --- Forecasting and Control: Revised Edition,\/} ISBN 0-8162-1104-3, Holden-Day, San Francisco, 1976.\ \ Jean-Ren\'e Barra, {\it Mathematical Basis of Statistics}, ISBN 0-12-079240-0, Academic Press, New York, 1981.\ \ So, what is really significant recent ML is adding to this quite old material? How much of the good stuff in that material has CS AI/ML even read and understood so far? Uh, what am I missing? I can think of a little but not much. CS AI/ML going for applications of statistics is a lot like Stanley Tools, long in the hammer, screwdriver, saw, drill, and nail gun business, going into the residential construction business building, selling new houses and claiming that they have something new! That is, because they make good nail guns they conclude that they are the best at building houses. They know nothing of excavation, masonry, framing carpentry, windows, doors, roofing, plumbing, electrical, HVAC, dry wall, painting, circular stairs, walnut paneling, landscaping, or even the building codes, but they've got some good nail guns! The math I derived and published in multi-dimensional, distribution-free anomaly detection has next to nothing to do with any of those books above; instead, I did original work and drew from ergodic theory, abstract algebra, and a classic result of Ulam. For the math for my startup, it's also original and even farther from those books above. In both cases, the core results are presented as theorems with proofs. Such math is mostly not what I'm seeing in CS AI/ML. I DID see some cute, recent work in analysis, maybe from ML, of the over fitting problem; I don't need that work for my startup, but I do intend to go back and read that work. My main points are: (1) If AI/ML have some applied math tools that are solid, good, and new, terrific. Then those cases will add to the many thousands of such tools already on the shelves of the research libraries. (2) The most important paradigm for a solid future for such tools is new, solid, correct, and powerful theorems and proofs building on solid material in math. That is, we want better stuff than is already in the libraries.
- srean 9y ago
  Agree with the gist but with some reservations. Small correction to what I said: srean> I was deliberately harsh on stats in my comment to offset some of the wrong things you mentioned about stats. Replace 'stats' with ML in my comment above. You are spot on regarding stats part, regarding ML not so much. In fact I have great respect for you as a applied mathematician and frequently learn something from these exchange of comments. graycat> Moreover, the situation is reversed: Columbus brought better ship, etc. technology to the Americas than the Americas had, but CS AL/ML are bringing nearly all highly inferior technology to statistics, pure/applied math, etc. This is unfair to the point of being absurd. Have you looked at stats software ? or compute infrastructure built by statisticians ? I can say more, but that would be below the belt. CS/ML has infused new blood in this area augmenting the capabilities of what can be done on a machine or cloud) 3 orders of magnitude or more. Part of it is advances in CS and software systems, but that is not all. One of the other parts is advances in algorithms. I am sure you would not scoff at ML pedigreed algorithms that run way faster than CPLEX. Many would not have the context, but I am sure you do. Of course these algorithms are not fr any LP, but a class of huge LPs that show up in graphical models. These belief propagation based algorithms outperform not only vanilla simplex algorithms but also the state of the at ones on these problems. Then there is culture. ML algorithms are more likely to be open sourced so that people try it, criticize it and that is how progress is made. In stats locking it behind some proprietary stuff is the norm. R is an exception, but as a piece of software I would keep it light years away from the money. Let me move on. graycat> The asymptotic stuff is mostly okay: Can't figure out precisely what the darned thing will do for case n so let n go to infinity and maybe can see what happens. Then say, "for large n, this is about what happens" -- crude but better than nothing. Yeah n is indeed very large in most data sets, but p the dimension is often larger. All hell breaks lose in this situation when you go by classic theory of statistical inference. This quandary applies to deep neural nets also. The number of parameters way exceeds the training data points it yet generalizes well to unseen data. WTF is going on ! Classic prob/stats have no answers. Forget classic, even modern prob stats is struggling to explain this. A good fight is being fought to characterize their behavior. New mathematical arguments are being developed but this is clearly a work in progress, an open problem. You have a chip on your shoulder of sorts regarding ML. That's fine, nothing wrong with that. What is wrong, however, is when you mischaracterize it because you are not reading the right material. You have to read those books and journal publications that are the moral equivalent of the prob/stats library you have. Its not fair to compare two languages when you quote from corny porn in one but high literature from the other. May I say Vapnik again :) And checkout the main publication s from COLT (computational learning theory), ICML, NIPS etc. I am sure you have heard of the multi-armed bandit problem. Gittins index and all that. Now solve it for the distribution free case. Even that leaves something to be desired because rewards may not be governed by a stochastic process. Give guarantees for arbitrary but bounded reward sequences. Stats literature had no answer for these cases. These are just a glimpse of the tools and results that have come from ML. You observe some noisy entries from a huge matrix that is low-rank. Now fill the unseen entries and give guarantees on performance. ML has the tools here. A biggie is analysis of causality thanks to Judea Pearl. Some still dont get it that conditioning and causality are fundamentally different. Just regressing Y over X will not cut it. ML has the tools here. Anyways... Oh Jesus! not ANOVA again. Its a cute toy, that's all. The assumptions it needs are rarely met in the datasets we want to analyze today. Visionaries like Tukey had figured that out 3 decades ago. Why do statisticians to this day keep using parametric tools in situations where we are drowning in data. I agree that its not entirely the fault of statistics that people do the above. Stats really has to shed a lot of baggage that made sense a century ago but does not make sense now. Given your experience I am sure you know that things are rarely ever Gaussian. Central limit theorem often does not apply in this popular form because of distributions with ginormous variance. Why use tools based on that. Classic parameter estimation guarantees are all fine in the toy 'spherical cow' world but where in stats are the non-asymptotic guarantees on prediction performance in a distribution free setting. This is precisely the raison d'être for ML. Not that there are no statisticians who have gone this route (Dawid would be one who needs mention), but way too few. ML does indeed have a lot to offer. To be cute I would leave it at what I said before: > non-asymptotic guarantees on prediction in a distribution free / nonparametric setting. That is what ML does. The newer developments aim at situations where the number of parameters in the model far exceed the data. What guarantees can you give in such situations. Surprisingly quite a bit, and contrary to what text books have said over the century, you can give guarantees. The algorithms work magically well, but of course you would need some (realistic assumptions). ML is an applied science so it borrows from and (crucially) adds to mathematics all the time. This particular line of work borrows a lot from the geometry of high dimensional random convex bodies, and Banach spaces. The most exciting part I see is the melding of the good parts of ML and stats and a good deal of communication across the Stats and ML community lines. They collaborate, they publish in the same conferences and that's just freaking fantastic. This is what we need not some stupid fights over which is better or is it applied science or not. Parting word, dont get an idea of ML from ML porn and spread untruths like, graycat> ML is .... short on math guarantees. build a library just like you did for stats.

by srean 9y ago

graycat 9y ago

I agree with a lot of that. But hypothesis testing remains important because in some contexts with too little data that's about the best can do. For parametric tests, there are some cases when that's okay. The asymptotic stuff is mostly okay: Can't figure out precisely what the darned thing will do for case n so let n go to infinity and maybe can see what happens. Then say, "for large n, this is about what happens" -- crude but better than nothing. The central limit theorem and more says that the Gaussian is really important. Okay. Done. We know that. But, right, for too long too much of statistics made nearly a religion out of the Gaussian. Bummer. Today happened to see the discussion panel at Stanford at https://www.youtube.com/watch?v=hxXIJnjC_HI https://www.youtube.com/watch?v=hxXIJnjC_HI It's fun stuff, with some big names. They are laughing at ML: ML discovered statistics like Columbus discovered America. Columbus didn't discover America since when he landed millions of people were already living there. And when CS AL/ML discovered statistics, there was already a huge field there. Moreover, the situation is reversed: Columbus brought better ship, etc. technology to the Americas than the Americas had, but CS AL/ML are bringing nearly all highly inferior technology to statistics, pure/applied math, etc. CS AI/ML is nearly all new labels for adulterated old wine. So, take some data on two variables, plot on an X-Y graph, fit a straight line, get the coefficients of the line as in first year algebra, and, presto, bingo, "The AI revolution is here!!!!", CS had had a machine "learn" the coefficients and they started with "training" data, that is trained the machine!!!!!! Where can I get one of those airline seat barf bags? Upchuck time! Uh, I have a pretty good professional library; much of my house is lined with -bookshelves. The main topics are pure/applied math. Some of the books relevant to statistics include (with TeX markup) Alexander M.\ Mood, Franklin A.\ Graybill, and Duane C.\ Boas, {\it Introduction to the Theory of Statistics, Third Edition,\/} McGraw-Hill, New York, 1974.\ \ N.\ R.\ Draper and H.\ Smith, {\it Applied Regression Analysis,\/} John Wiley and Sons, New York, 1968.\ \ C.\ Radhakrishna Rao, {\it Linear Statistical Inference and Its Applications:\ \ Second Edition,\/} ISBN 0-471-70823-2, John Wiley and Sons, New York, 1967.\ \ Henry Scheff\'e, {\it Analysis of Variance,\/} John Wiley and Sons, New York, 1967.\ \ Yvonne M.\ M.\ Bishop, Stephen E.\ Fienberg, Paul W.\ Holland, {\it Discrete Multivariate Analysis:\ \ Theory and Practice,\/} ISBN 0-262-52040-0, MIT Press, Cambridge, Massachusetts, 1979.\ \ Stephen E.\ Fienberg, {\it The Analysis of Cross-Classified Data,\/} ISBN 0-262-06063-9, MIT Press, Cambridge, Massachusetts, 1979.\ \ Leo Breiman, Jerome H.\ Friedman, Richard A.\ Olshen, Charles J.\ Stone, {\it Classification and Regression Trees,\/} ISBN 0-534-98054-6, Wadsworth \& Brooks/Cole, Pacific Grove, California, 1984.\ \ R.\ B.\ Blackman and J.\ W.\ Tukey, {\it The Measurement of Power Spectra:\ \ From the Point of View of Communications Engineering,\/} Dover, New York, 1959.\ \ William W.\ Cooley and Paul R.\ Lohnes, {\it Multivariate Data Analysis,\/} John Wiley and Sons, New York, 1971.\ \ Maurice M.\ Tatsuoka, {\it Multivariate Analysis: Techniques for Educational and Psychological Research,\/} John Wiley and Sons, 1971.\ \ E.\ L.\ Lehmann, {\it Testing Statistical Hypotheses,\/} John Wiley, New York, 1959.\ \ E.\ L.\ Lehmann, {\it Nonparametrics: Statistical Methods Based on Ranks,\/} ISBN 0-8162-4994-6, Holden-Day, San Francisco, 1975.\ \ Jaroslav H\'ajek and Zbyn\v ek \v Sid\'ak, {\it Theory of Rank Tests,\/} Academia, Prague, 1967.\ \ Sidney Siegel, {\it Nonparametric Statistics for the Behavioral Sciences,\/} McGraw-Hill, New York, 1956.\ \ Donald F.\ Morrison, {\it Multivariate Statistical Methods: Second Edition,\/} ISBN 0-07-043186-8, McGraw-Hill, New York, 1976.\ \ Harry H.\ Harman, {\it Modern Factor Analysis: Second Edition, Revised,\/} The University of Chicago Press, Chicago, 1967.\ \ George E.\ P.\ Box and Gwilym M.\ Jenkins, {\it Time Series Analysis --- Forecasting and Control: Revised Edition,\/} ISBN 0-8162-1104-3, Holden-Day, San Francisco, 1976.\ \ Jean-Ren\'e Barra, {\it Mathematical Basis of Statistics}, ISBN 0-12-079240-0, Academic Press, New York, 1981.\ \ So, what is really significant recent ML is adding to this quite old material? How much of the good stuff in that material has CS AI/ML even read and understood so far? Uh, what am I missing? I can think of a little but not much. CS AI/ML going for applications of statistics is a lot like Stanley Tools, long in the hammer, screwdriver, saw, drill, and nail gun business, going into the residential construction business building, selling new houses and claiming that they have something new! That is, because they make good nail guns they conclude that they are the best at building houses. They know nothing of excavation, masonry, framing carpentry, windows, doors, roofing, plumbing, electrical, HVAC, dry wall, painting, circular stairs, walnut paneling, landscaping, or even the building codes, but they've got some good nail guns! The math I derived and published in multi-dimensional, distribution-free anomaly detection has next to nothing to do with any of those books above; instead, I did original work and drew from ergodic theory, abstract algebra, and a classic result of Ulam. For the math for my startup, it's also original and even farther from those books above. In both cases, the core results are presented as theorems with proofs. Such math is mostly not what I'm seeing in CS AI/ML. I DID see some cute, recent work in analysis, maybe from ML, of the over fitting problem; I don't need that work for my startup, but I do intend to go back and read that work. My main points are: (1) If AI/ML have some applied math tools that are solid, good, and new, terrific. Then those cases will add to the many thousands of such tools already on the shelves of the research libraries. (2) The most important paradigm for a solid future for such tools is new, solid, correct, and powerful theorems and proofs building on solid material in math. That is, we want better stuff than is already in the libraries.

srean 9y ago
Agree with the gist but with some reservations. Small correction to what I said: srean> I was deliberately harsh on stats in my comment to offset some of the wrong things you mentioned about stats. Replace 'stats' with ML in my comment above. You are spot on regarding stats part, regarding ML not so much. In fact I have great respect for you as a applied mathematician and frequently learn something from these exchange of comments. graycat> Moreover, the situation is reversed: Columbus brought better ship, etc. technology to the Americas than the Americas had, but CS AL/ML are bringing nearly all highly inferior technology to statistics, pure/applied math, etc. This is unfair to the point of being absurd. Have you looked at stats software ? or compute infrastructure built by statisticians ? I can say more, but that would be below the belt. CS/ML has infused new blood in this area augmenting the capabilities of what can be done on a machine or cloud) 3 orders of magnitude or more. Part of it is advances in CS and software systems, but that is not all. One of the other parts is advances in algorithms. I am sure you would not scoff at ML pedigreed algorithms that run way faster than CPLEX. Many would not have the context, but I am sure you do. Of course these algorithms are not fr any LP, but a class of huge LPs that show up in graphical models. These belief propagation based algorithms outperform not only vanilla simplex algorithms but also the state of the at ones on these problems. Then there is culture. ML algorithms are more likely to be open sourced so that people try it, criticize it and that is how progress is made. In stats locking it behind some proprietary stuff is the norm. R is an exception, but as a piece of software I would keep it light years away from the money. Let me move on. graycat> The asymptotic stuff is mostly okay: Can't figure out precisely what the darned thing will do for case n so let n go to infinity and maybe can see what happens. Then say, "for large n, this is about what happens" -- crude but better than nothing. Yeah n is indeed very large in most data sets, but p the dimension is often larger. All hell breaks lose in this situation when you go by classic theory of statistical inference. This quandary applies to deep neural nets also. The number of parameters way exceeds the training data points it yet generalizes well to unseen data. WTF is going on ! Classic prob/stats have no answers. Forget classic, even modern prob stats is struggling to explain this. A good fight is being fought to characterize their behavior. New mathematical arguments are being developed but this is clearly a work in progress, an open problem. You have a chip on your shoulder of sorts regarding ML. That's fine, nothing wrong with that. What is wrong, however, is when you mischaracterize it because you are not reading the right material. You have to read those books and journal publications that are the moral equivalent of the prob/stats library you have. Its not fair to compare two languages when you quote from corny porn in one but high literature from the other. May I say Vapnik again :) And checkout the main publication s from COLT (computational learning theory), ICML, NIPS etc. I am sure you have heard of the multi-armed bandit problem. Gittins index and all that. Now solve it for the distribution free case. Even that leaves something to be desired because rewards may not be governed by a stochastic process. Give guarantees for arbitrary but bounded reward sequences. Stats literature had no answer for these cases. These are just a glimpse of the tools and results that have come from ML. You observe some noisy entries from a huge matrix that is low-rank. Now fill the unseen entries and give guarantees on performance. ML has the tools here. A biggie is analysis of causality thanks to Judea Pearl. Some still dont get it that conditioning and causality are fundamentally different. Just regressing Y over X will not cut it. ML has the tools here. Anyways... Oh Jesus! not ANOVA again. Its a cute toy, that's all. The assumptions it needs are rarely met in the datasets we want to analyze today. Visionaries like Tukey had figured that out 3 decades ago. Why do statisticians to this day keep using parametric tools in situations where we are drowning in data. I agree that its not entirely the fault of statistics that people do the above. Stats really has to shed a lot of baggage that made sense a century ago but does not make sense now. Given your experience I am sure you know that things are rarely ever Gaussian. Central limit theorem often does not apply in this popular form because of distributions with ginormous variance. Why use tools based on that. Classic parameter estimation guarantees are all fine in the toy 'spherical cow' world but where in stats are the non-asymptotic guarantees on prediction performance in a distribution free setting. This is precisely the raison d'être for ML. Not that there are no statisticians who have gone this route (Dawid would be one who needs mention), but way too few. ML does indeed have a lot to offer. To be cute I would leave it at what I said before: > non-asymptotic guarantees on prediction in a distribution free / nonparametric setting. That is what ML does. The newer developments aim at situations where the number of parameters in the model far exceed the data. What guarantees can you give in such situations. Surprisingly quite a bit, and contrary to what text books have said over the century, you can give guarantees. The algorithms work magically well, but of course you would need some (realistic assumptions). ML is an applied science so it borrows from and (crucially) adds to mathematics all the time. This particular line of work borrows a lot from the geometry of high dimensional random convex bodies, and Banach spaces. The most exciting part I see is the melding of the good parts of ML and stats and a good deal of communication across the Stats and ML community lines. They collaborate, they publish in the same conferences and that's just freaking fantastic. This is what we need not some stupid fights over which is better or is it applied science or not. Parting word, dont get an idea of ML from ML porn and spread untruths like, graycat> ML is .... short on math guarantees. build a library just like you did for stats.