7 ms·

You’ve told us vague things about wide areas of mathematics, but you haven’t told us what problem your start-up is solving using these branches of mathematics o

You’ve told us vague things about wide areas of mathematics, but you haven’t told us what problem your start-up is solving using these branches of mathematics or how it is a better allocation of investor dollars than focusing on ML. I’ve worked on a wide range of applied mathematics problems in machine learning, computer vision, signal / image processing, control theory, and applied statistics. Very few people outside of a mathematics department care at all about theorems, proofs, and "correctness" in the formal mathematical sense. I can count on one hand how many times I’ve needed to actually do a proof, and the number of people who have cared is smaller still. All anyone cares about is how well you can solve their problems. Sometimes advanced mathematics is the right tool for the job, but often it is not.

graycat 9y ago
> You’ve told us vague things about wide areas of mathematics, but you haven’t told us what problem your start-up is solving using these branches of mathematics or how it is a better allocation of investor dollars than focusing on ML. Easy: ML is quite narrow. More options yield the same, usually better, investment results. Compared with the applied math on the shelves of the libraries, ML is nearly pinpoint narrow. In addition, too much of ML is heuristic curve fitting short on math guarantees. Also, where some real problems have some features that justify some math assumptions, ML is slow to exploit those. In the workshop, kitchen, or lab, use the best tools you can. So don't limit yourself to just what saw on a cooking show on TV. My view is that mathematical proof is a fantastically powerful part of applying math. The core reason is that it lets do some additional, maybe significantly original, derivations to get better solutions for real problems. E.g., once I was in an AI group working on monitoring and managing server farms and networks. We got rivers of data on many variables at high data rates. There we wanted to detect problems ASAP in near real time. Okay, necessarily, inescapably, we have two ways to be wrong -- false positives (claiming the system is sick when it's healthy) and false negatives (claiming the systems is healthy when it's sick). Okay, then, right away we are forced, like it or not, into the framework of some statistical hypothesis tests continually applied. There, as is usual in hypothesis testing, we want to be able to select a false alarm rate and get that rate exactly. We'd like the most powerful test as in the lemma of Neyman-Pearson but likely do not have enough data. Still we want test power in some good sense; if we are willing to tolerate a higher false alarm rate, then we'd like a higher detection rate; when we get a detection, we'd like to know, as a measure of seriousness, the lowest false alarm rate at which the data would still be a detection; at least we don't want a trivial test. For this data, no way can we know the probability distribution of the data under the null hypothesis (the system is healthy). This is because even the univariate data is commonly a total mess, far worse than was long common in statistics (U. Grenander explained this to me in his office at Brown). And, of course, with so much data on so many variables, we should be doing multi-variate testing. So, net we need tests that are both multi-variate and distribution-free. Go look for those in E. Lehmann, etc. and won't find any. So, I invented some. Then, how the heck to adjust false alarm rate? That was a problem in applied probability that needed new theorems and proofs. So, I did those. For those I used measure theory, measure preserving from ergodic theory, group theory from abstract algebra, and more. Yup, theorems and proofs. Otherwise I would not have known what the heck I had, what the properties were, what the false alarm rates were, etc. Then do my tests have any power? I didn't have data enough to apply Neyman-Pearson, but I found a way. Are my tests trivial? I used a result of S. Ulam and stated and proved a theorem -- not trivial. The work was intended to be practical. Likely and apparently the work remains the cat's meow, best in the world, for detection of problems never seen before -- behavioral monitoring, zero day monitoring. Gee, just read that the last security attack on Target cost them $300 million. They needed better monitoring! New theorems and proofs were crucial, and the work was intended to be practical. A guy had a fast opportunity to do some marketing but, with the short time interval, had limited resources. He had a 0-1 linear programming formulation, 40,000 constraints and 600,000 variables for a small, test case. Sure, no linear programming or integer linear programming package would have a chance. So, I got his formulation, saw some special structure to exploit, did some non-linear duality derivations for some Lagrangian relaxation, found how to get some bounds, wrote some code, and in 905 seconds on a slow computer found feasible solutions guaranteed to be within 0.025% of optimality. Well, the derivations were essentially theorems and proofs. He'd tried a computer science approach, simulated annealing. He ran for days and then stopped with the best he had and with no idea how close he was to optimality. My work, from theorems and proofs, was much, MUCH better. The crucial core applied math of my startup is some theorems and proofs; I wouldn't believe that the mathematical operations would yield what I want otherwise. Sure, usually a working applied mathematician doesn't always have to stir up new theorems and proofs, but at times that's darned helpful and valuable.
- srean 9y ago
  Hey sorry but this > In addition, too much of ML is .... short on math guarantees. is utterly false. As I keep telling you need to upgrade :) In fact I will level the same exact criticism on standard stats. NP lemma is fantastic math but of little practical importance because you rarely ever know the densities. Most of the time 'test of hypothesis' is a bad joke catering to a contrived scenario. Real problems are rarely about 'is it H0 or H1'. Estimation is typically more useful. Stats have long suffered from their fetish for parametric models, asymptotic normality, asymptotic guarantees and what i feel their greatest deficiency: the focus on parameter estimation rather than guarantees over prediction. Parameters and parametric models are convenient fiction, nobodody has seen them, will ever see them. What's all the fuss about estimating a fictional object to great accuracy under strong assumptions in infinite time about. Made sense a century ago though. If their purported utility is in aiding prediction then why not go after the real deal, that is, non-asymptotic guarantees on prediction in a distribution free / nonparametric setting. That is what ML does. Please dont characterize ML by bad examples and work in progress stuff. I do know nonparametric stats exist but thats not the face of stats that people get to see, but even in those cases the focus is usually on accuracy of estimating a functional. I was deliberately harsh on stats in my comment to offset some of the wrong things you mentioned about stats, but the relation between ml and stats are really not adversarial. I see ML as a course correction for stats: focus on prediction guarantees and adding some muscle in two areas (i) algorithms for scaling (ii) optimization. In anycase i know where both the tribe hide their dirty laundry :) That said I strongly agree that Ml is indeed applied math a mix of probability, optimization, functional analysis and algorithms/Datastructures
  - graycat 9y ago
    I agree with a lot of that. But hypothesis testing remains important because in some contexts with too little data that's about the best can do. For parametric tests, there are some cases when that's okay. The asymptotic stuff is mostly okay: Can't figure out precisely what the darned thing will do for case n so let n go to infinity and maybe can see what happens. Then say, "for large n, this is about what happens" -- crude but better than nothing. The central limit theorem and more says that the Gaussian is really important. Okay. Done. We know that. But, right, for too long too much of statistics made nearly a religion out of the Gaussian. Bummer. Today happened to see the discussion panel at Stanford at https://www.youtube.com/watch?v=hxXIJnjC_HI https://www.youtube.com/watch?v=hxXIJnjC_HI It's fun stuff, with some big names. They are laughing at ML: ML discovered statistics like Columbus discovered America. Columbus didn't discover America since when he landed millions of people were already living there. And when CS AL/ML discovered statistics, there was already a huge field there. Moreover, the situation is reversed: Columbus brought better ship, etc. technology to the Americas than the Americas had, but CS AL/ML are bringing nearly all highly inferior technology to statistics, pure/applied math, etc. CS AI/ML is nearly all new labels for adulterated old wine. So, take some data on two variables, plot on an X-Y graph, fit a straight line, get the coefficients of the line as in first year algebra, and, presto, bingo, "The AI revolution is here!!!!", CS had had a machine "learn" the coefficients and they started with "training" data, that is trained the machine!!!!!! Where can I get one of those airline seat barf bags? Upchuck time! Uh, I have a pretty good professional library; much of my house is lined with -bookshelves. The main topics are pure/applied math. Some of the books relevant to statistics include (with TeX markup) Alexander M.\ Mood, Franklin A.\ Graybill, and Duane C.\ Boas, {\it Introduction to the Theory of Statistics, Third Edition,\/} McGraw-Hill, New York, 1974.\ \ N.\ R.\ Draper and H.\ Smith, {\it Applied Regression Analysis,\/} John Wiley and Sons, New York, 1968.\ \ C.\ Radhakrishna Rao, {\it Linear Statistical Inference and Its Applications:\ \ Second Edition,\/} ISBN 0-471-70823-2, John Wiley and Sons, New York, 1967.\ \ Henry Scheff\'e, {\it Analysis of Variance,\/} John Wiley and Sons, New York, 1967.\ \ Yvonne M.\ M.\ Bishop, Stephen E.\ Fienberg, Paul W.\ Holland, {\it Discrete Multivariate Analysis:\ \ Theory and Practice,\/} ISBN 0-262-52040-0, MIT Press, Cambridge, Massachusetts, 1979.\ \ Stephen E.\ Fienberg, {\it The Analysis of Cross-Classified Data,\/} ISBN 0-262-06063-9, MIT Press, Cambridge, Massachusetts, 1979.\ \ Leo Breiman, Jerome H.\ Friedman, Richard A.\ Olshen, Charles J.\ Stone, {\it Classification and Regression Trees,\/} ISBN 0-534-98054-6, Wadsworth \& Brooks/Cole, Pacific Grove, California, 1984.\ \ R.\ B.\ Blackman and J.\ W.\ Tukey, {\it The Measurement of Power Spectra:\ \ From the Point of View of Communications Engineering,\/} Dover, New York, 1959.\ \ William W.\ Cooley and Paul R.\ Lohnes, {\it Multivariate Data Analysis,\/} John Wiley and Sons, New York, 1971.\ \ Maurice M.\ Tatsuoka, {\it Multivariate Analysis: Techniques for Educational and Psychological Research,\/} John Wiley and Sons, 1971.\ \ E.\ L.\ Lehmann, {\it Testing Statistical Hypotheses,\/} John Wiley, New York, 1959.\ \ E.\ L.\ Lehmann, {\it Nonparametrics: Statistical Methods Based on Ranks,\/} ISBN 0-8162-4994-6, Holden-Day, San Francisco, 1975.\ \ Jaroslav H\'ajek and Zbyn\v ek \v Sid\'ak, {\it Theory of Rank Tests,\/} Academia, Prague, 1967.\ \ Sidney Siegel, {\it Nonparametric Statistics for the Behavioral Sciences,\/} McGraw-Hill, New York, 1956.\ \ Donald F.\ Morrison, {\it Multivariate Statistical Methods: Second Edition,\/} ISBN 0-07-043186-8, McGraw-Hill, New York, 1976.\ \ Harry H.\ Harman, {\it Modern Factor Analysis: Second Edition, Revised,\/} The University of Chicago Press, Chicago, 1967.\ \ George E.\ P.\ Box and Gwilym M.\ Jenkins, {\it Time Series Analysis --- Forecasting and Control: Revised Edition,\/} ISBN 0-8162-1104-3, Holden-Day, San Francisco, 1976.\ \ Jean-Ren\'e Barra, {\it Mathematical Basis of Statistics}, ISBN 0-12-079240-0, Academic Press, New York, 1981.\ \ So, what is really significant recent ML is adding to this quite old material? How much of the good stuff in that material has CS AI/ML even read and understood so far? Uh, what am I missing? I can think of a little but not much. CS AI/ML going for applications of statistics is a lot like Stanley Tools, long in the hammer, screwdriver, saw, drill, and nail gun business, going into the residential construction business building, selling new houses and claiming that they have something new! That is, because they make good nail guns they conclude that they are the best at building houses. They know nothing of excavation, masonry, framing carpentry, windows, doors, roofing, plumbing, electrical, HVAC, dry wall, painting, circular stairs, walnut paneling, landscaping, or even the building codes, but they've got some good nail guns! The math I derived and published in multi-dimensional, distribution-free anomaly detection has next to nothing to do with any of those books above; instead, I did original work and drew from ergodic theory, abstract algebra, and a classic result of Ulam. For the math for my startup, it's also original and even farther from those books above. In both cases, the core results are presented as theorems with proofs. Such math is mostly not what I'm seeing in CS AI/ML. I DID see some cute, recent work in analysis, maybe from ML, of the over fitting problem; I don't need that work for my startup, but I do intend to go back and read that work. My main points are: (1) If AI/ML have some applied math tools that are solid, good, and new, terrific. Then those cases will add to the many thousands of such tools already on the shelves of the research libraries. (2) The most important paradigm for a solid future for such tools is new, solid, correct, and powerful theorems and proofs building on solid material in math. That is, we want better stuff than is already in the libraries.

graycat 9y ago

> You’ve told us vague things about wide areas of mathematics, but you haven’t told us what problem your start-up is solving using these branches of mathematics or how it is a better allocation of investor dollars than focusing on ML. Easy: ML is quite narrow. More options yield the same, usually better, investment results. Compared with the applied math on the shelves of the libraries, ML is nearly pinpoint narrow. In addition, too much of ML is heuristic curve fitting short on math guarantees. Also, where some real problems have some features that justify some math assumptions, ML is slow to exploit those. In the workshop, kitchen, or lab, use the best tools you can. So don't limit yourself to just what saw on a cooking show on TV. My view is that mathematical proof is a fantastically powerful part of applying math. The core reason is that it lets do some additional, maybe significantly original, derivations to get better solutions for real problems. E.g., once I was in an AI group working on monitoring and managing server farms and networks. We got rivers of data on many variables at high data rates. There we wanted to detect problems ASAP in near real time. Okay, necessarily, inescapably, we have two ways to be wrong -- false positives (claiming the system is sick when it's healthy) and false negatives (claiming the systems is healthy when it's sick). Okay, then, right away we are forced, like it or not, into the framework of some statistical hypothesis tests continually applied. There, as is usual in hypothesis testing, we want to be able to select a false alarm rate and get that rate exactly. We'd like the most powerful test as in the lemma of Neyman-Pearson but likely do not have enough data. Still we want test power in some good sense; if we are willing to tolerate a higher false alarm rate, then we'd like a higher detection rate; when we get a detection, we'd like to know, as a measure of seriousness, the lowest false alarm rate at which the data would still be a detection; at least we don't want a trivial test. For this data, no way can we know the probability distribution of the data under the null hypothesis (the system is healthy). This is because even the univariate data is commonly a total mess, far worse than was long common in statistics (U. Grenander explained this to me in his office at Brown). And, of course, with so much data on so many variables, we should be doing multi-variate testing. So, net we need tests that are both multi-variate and distribution-free. Go look for those in E. Lehmann, etc. and won't find any. So, I invented some. Then, how the heck to adjust false alarm rate? That was a problem in applied probability that needed new theorems and proofs. So, I did those. For those I used measure theory, measure preserving from ergodic theory, group theory from abstract algebra, and more. Yup, theorems and proofs. Otherwise I would not have known what the heck I had, what the properties were, what the false alarm rates were, etc. Then do my tests have any power? I didn't have data enough to apply Neyman-Pearson, but I found a way. Are my tests trivial? I used a result of S. Ulam and stated and proved a theorem -- not trivial. The work was intended to be practical. Likely and apparently the work remains the cat's meow, best in the world, for detection of problems never seen before -- behavioral monitoring, zero day monitoring. Gee, just read that the last security attack on Target cost them $300 million. They needed better monitoring! New theorems and proofs were crucial, and the work was intended to be practical. A guy had a fast opportunity to do some marketing but, with the short time interval, had limited resources. He had a 0-1 linear programming formulation, 40,000 constraints and 600,000 variables for a small, test case. Sure, no linear programming or integer linear programming package would have a chance. So, I got his formulation, saw some special structure to exploit, did some non-linear duality derivations for some Lagrangian relaxation, found how to get some bounds, wrote some code, and in 905 seconds on a slow computer found feasible solutions guaranteed to be within 0.025% of optimality. Well, the derivations were essentially theorems and proofs. He'd tried a computer science approach, simulated annealing. He ran for days and then stopped with the best he had and with no idea how close he was to optimality. My work, from theorems and proofs, was much, MUCH better. The crucial core applied math of my startup is some theorems and proofs; I wouldn't believe that the mathematical operations would yield what I want otherwise. Sure, usually a working applied mathematician doesn't always have to stir up new theorems and proofs, but at times that's darned helpful and valuable.

srean 9y ago
Hey sorry but this > In addition, too much of ML is .... short on math guarantees. is utterly false. As I keep telling you need to upgrade :) In fact I will level the same exact criticism on standard stats. NP lemma is fantastic math but of little practical importance because you rarely ever know the densities. Most of the time 'test of hypothesis' is a bad joke catering to a contrived scenario. Real problems are rarely about 'is it H0 or H1'. Estimation is typically more useful. Stats have long suffered from their fetish for parametric models, asymptotic normality, asymptotic guarantees and what i feel their greatest deficiency: the focus on parameter estimation rather than guarantees over prediction. Parameters and parametric models are convenient fiction, nobodody has seen them, will ever see them. What's all the fuss about estimating a fictional object to great accuracy under strong assumptions in infinite time about. Made sense a century ago though. If their purported utility is in aiding prediction then why not go after the real deal, that is, non-asymptotic guarantees on prediction in a distribution free / nonparametric setting. That is what ML does. Please dont characterize ML by bad examples and work in progress stuff. I do know nonparametric stats exist but thats not the face of stats that people get to see, but even in those cases the focus is usually on accuracy of estimating a functional. I was deliberately harsh on stats in my comment to offset some of the wrong things you mentioned about stats, but the relation between ml and stats are really not adversarial. I see ML as a course correction for stats: focus on prediction guarantees and adding some muscle in two areas (i) algorithms for scaling (ii) optimization. In anycase i know where both the tribe hide their dirty laundry :) That said I strongly agree that Ml is indeed applied math a mix of probability, optimization, functional analysis and algorithms/Datastructures
- graycat 9y ago
  I agree with a lot of that. But hypothesis testing remains important because in some contexts with too little data that's about the best can do. For parametric tests, there are some cases when that's okay. The asymptotic stuff is mostly okay: Can't figure out precisely what the darned thing will do for case n so let n go to infinity and maybe can see what happens. Then say, "for large n, this is about what happens" -- crude but better than nothing. The central limit theorem and more says that the Gaussian is really important. Okay. Done. We know that. But, right, for too long too much of statistics made nearly a religion out of the Gaussian. Bummer. Today happened to see the discussion panel at Stanford at https://www.youtube.com/watch?v=hxXIJnjC_HI https://www.youtube.com/watch?v=hxXIJnjC_HI It's fun stuff, with some big names. They are laughing at ML: ML discovered statistics like Columbus discovered America. Columbus didn't discover America since when he landed millions of people were already living there. And when CS AL/ML discovered statistics, there was already a huge field there. Moreover, the situation is reversed: Columbus brought better ship, etc. technology to the Americas than the Americas had, but CS AL/ML are bringing nearly all highly inferior technology to statistics, pure/applied math, etc. CS AI/ML is nearly all new labels for adulterated old wine. So, take some data on two variables, plot on an X-Y graph, fit a straight line, get the coefficients of the line as in first year algebra, and, presto, bingo, "The AI revolution is here!!!!", CS had had a machine "learn" the coefficients and they started with "training" data, that is trained the machine!!!!!! Where can I get one of those airline seat barf bags? Upchuck time! Uh, I have a pretty good professional library; much of my house is lined with -bookshelves. The main topics are pure/applied math. Some of the books relevant to statistics include (with TeX markup) Alexander M.\ Mood, Franklin A.\ Graybill, and Duane C.\ Boas, {\it Introduction to the Theory of Statistics, Third Edition,\/} McGraw-Hill, New York, 1974.\ \ N.\ R.\ Draper and H.\ Smith, {\it Applied Regression Analysis,\/} John Wiley and Sons, New York, 1968.\ \ C.\ Radhakrishna Rao, {\it Linear Statistical Inference and Its Applications:\ \ Second Edition,\/} ISBN 0-471-70823-2, John Wiley and Sons, New York, 1967.\ \ Henry Scheff\'e, {\it Analysis of Variance,\/} John Wiley and Sons, New York, 1967.\ \ Yvonne M.\ M.\ Bishop, Stephen E.\ Fienberg, Paul W.\ Holland, {\it Discrete Multivariate Analysis:\ \ Theory and Practice,\/} ISBN 0-262-52040-0, MIT Press, Cambridge, Massachusetts, 1979.\ \ Stephen E.\ Fienberg, {\it The Analysis of Cross-Classified Data,\/} ISBN 0-262-06063-9, MIT Press, Cambridge, Massachusetts, 1979.\ \ Leo Breiman, Jerome H.\ Friedman, Richard A.\ Olshen, Charles J.\ Stone, {\it Classification and Regression Trees,\/} ISBN 0-534-98054-6, Wadsworth \& Brooks/Cole, Pacific Grove, California, 1984.\ \ R.\ B.\ Blackman and J.\ W.\ Tukey, {\it The Measurement of Power Spectra:\ \ From the Point of View of Communications Engineering,\/} Dover, New York, 1959.\ \ William W.\ Cooley and Paul R.\ Lohnes, {\it Multivariate Data Analysis,\/} John Wiley and Sons, New York, 1971.\ \ Maurice M.\ Tatsuoka, {\it Multivariate Analysis: Techniques for Educational and Psychological Research,\/} John Wiley and Sons, 1971.\ \ E.\ L.\ Lehmann, {\it Testing Statistical Hypotheses,\/} John Wiley, New York, 1959.\ \ E.\ L.\ Lehmann, {\it Nonparametrics: Statistical Methods Based on Ranks,\/} ISBN 0-8162-4994-6, Holden-Day, San Francisco, 1975.\ \ Jaroslav H\'ajek and Zbyn\v ek \v Sid\'ak, {\it Theory of Rank Tests,\/} Academia, Prague, 1967.\ \ Sidney Siegel, {\it Nonparametric Statistics for the Behavioral Sciences,\/} McGraw-Hill, New York, 1956.\ \ Donald F.\ Morrison, {\it Multivariate Statistical Methods: Second Edition,\/} ISBN 0-07-043186-8, McGraw-Hill, New York, 1976.\ \ Harry H.\ Harman, {\it Modern Factor Analysis: Second Edition, Revised,\/} The University of Chicago Press, Chicago, 1967.\ \ George E.\ P.\ Box and Gwilym M.\ Jenkins, {\it Time Series Analysis --- Forecasting and Control: Revised Edition,\/} ISBN 0-8162-1104-3, Holden-Day, San Francisco, 1976.\ \ Jean-Ren\'e Barra, {\it Mathematical Basis of Statistics}, ISBN 0-12-079240-0, Academic Press, New York, 1981.\ \ So, what is really significant recent ML is adding to this quite old material? How much of the good stuff in that material has CS AI/ML even read and understood so far? Uh, what am I missing? I can think of a little but not much. CS AI/ML going for applications of statistics is a lot like Stanley Tools, long in the hammer, screwdriver, saw, drill, and nail gun business, going into the residential construction business building, selling new houses and claiming that they have something new! That is, because they make good nail guns they conclude that they are the best at building houses. They know nothing of excavation, masonry, framing carpentry, windows, doors, roofing, plumbing, electrical, HVAC, dry wall, painting, circular stairs, walnut paneling, landscaping, or even the building codes, but they've got some good nail guns! The math I derived and published in multi-dimensional, distribution-free anomaly detection has next to nothing to do with any of those books above; instead, I did original work and drew from ergodic theory, abstract algebra, and a classic result of Ulam. For the math for my startup, it's also original and even farther from those books above. In both cases, the core results are presented as theorems with proofs. Such math is mostly not what I'm seeing in CS AI/ML. I DID see some cute, recent work in analysis, maybe from ML, of the over fitting problem; I don't need that work for my startup, but I do intend to go back and read that work. My main points are: (1) If AI/ML have some applied math tools that are solid, good, and new, terrific. Then those cases will add to the many thousands of such tools already on the shelves of the research libraries. (2) The most important paradigm for a solid future for such tools is new, solid, correct, and powerful theorems and proofs building on solid material in math. That is, we want better stuff than is already in the libraries.