5 ms·

An honest question - do we even need statistics when we have machine learning? Statistics to me appears as a hack/aggregation of data we couldn't process at onc

by bitL 10y ago

An honest question - do we even need statistics when we have machine learning? Statistics to me appears as a hack/aggregation of data we couldn't process at once in the past; these days ML + Big Data can achieve that and instead of statistics we can do computational inference instead. To me this looks like looking back to "old ways" for a reference point instead of looking forward to the unknown but more exciting.

Normal_gaussian 10y ago
ML + Big Data are a specific application of statistics To to do anything beyond use tools other people have made (and never be sure whether results are meaningful or not) statistics are required Of course, to make money from the ML boom you can probably get away with coincidence and correlation
- bitL 10y ago
  Statistics means aggregate stuff and uses simplified characteristics out of semi-structured data. ML + Big Data allows you to ask precise questions like Where? How? Which ones?
  - nfoz 10y ago
    As user "highd" suggested, I think you are confusing two words. I refer to Wiktionary for definitions: Statistics: A mathematical science concerned with data collection, presentation, analysis, and interpretation. Statistic: A quantity calculated from the data in a sample, which characterises an important aspect in the sample (such as mean or standard deviation). If "statistics" is the term for taking a mathematical approach to understanding data, then "machine learning" is basically an applied subset of that. But you seem to be specifically using the "a statistic" definition to describe what you think the "study of statistics" is entirely concerned with.
_Wintermute 10y ago
Yes we still need statistics. There is a huge overlap between machine learning methods and applied statistics, so much so that often there is not a clear distinction between the two.
- j1vms 10y ago
  > between machine learning methods and applied statistics (...) often there is not a clear distinction between the two. I would say applied statistics draws a line just prior to implementation concerns (say, real-world resource usage measured in time, space and energy) whereas these would be fully within scope and of interest in machine learning. As an example, applied statistics could provide a useful approach to a vision/image recognition problem, and this approach might be provably unrealizable in practice using real-world execution units (e.g. CUDA cores). Nonetheless, it might still be a very worthwhile theoretical result in applied statistics, although of no immediate interest within ML except to hint at potential new area of research.
joeyo 10y ago
It may not be true for all branches of machine learning (fuzzy logic, for example?), but the vast majority of modern ML techniques are equivalent to or can be viewed as types of statistical machine learning.
- Normal_gaussian 10y ago
  Good point about the fuzzy logic, often the boundaries between it and statistics are... fuzzy
klodolph 10y ago
Machine learning is often considered a statistical technique. The main difference seems to be that in traditional statistics, people derive practice from theory, whereas in ML people will try out techniques and figure out the theory later. That's really just a cultural difference. The techniques for analyzing ML models are all statistical to begin with. Statistics, as a field, already used general-purpose optimization algorithms before modern ML techniques came about, so in that sense, ML just fits into an existing position in the statistical toolbox (like replacing a chisel with a 3D printer). In the other direction, statistical techniques like cross-validation are necessary for you to get your ML correct.
- bitL 10y ago
  There is much more in ML than just statistics. I was basically asking why the "statistics filter" is so often on in ML. Neural networks don't seem a statistical technique, even if somebody uses them for regression. Yes, there is an overlap, but no, ML != statistics. As you mentioned, non-linear optimization is used in statistics on meta-level however nobody claims statistics is operations research or vice versa.
  - 443633456 10y ago
    I agree with this comment because ML comprises some techniques that are still not understood [1]. I would like to think that statistics comes more from a pure math approach, loosely, while ML comes from an applied math approach, loosely. ML works spectacularly well on a class of problems. Why it does what it does is (in)conveniently brushed under the rug. How you treat that (in)convenience is left to you. [1] - https://www.quantamagazine.org/20151203-big-datas-mathematical-mysteries/ https://www.quantamagazine.org/20151203-big-datas-mathematic...
  - srean 10y ago
    In what way is neural network not a statistical technique ? very curious to hear about your point of view. Statistics is not linear regression and ANOVA, or whatever catalogue of techniques in a freshman book, not even the library of techniques available in R. Statistics is the application of probability (or more broadly, math) to data. That said statisticians did miss the neural net wave because of their flippant reaction to it. They said, "oh well yet another non-parametric function approximator we have worked out the asymptotics 30 years ago". To paraphrase someone wise: asymptotically we are all dead. Not enough heed was paid to that. Among their other lacks were expertise in algorithms and optimization. Mind you, optimization has been at the core of their craft from their very genesis, its just that they did not feel it important enough to ride the cutting edge of research on optimization. Note: you cannot do maximum likelihood with solving an optimization problem. Gauss was doing it several hundreds of years ago for statistics. If I go on a bit further with my rant, they got a bit carried away with their fetish over bias and asymptotic normality. They missed the wave, sure. But all said and done, by any accepted definition of statistics, NN is very much also statistics.
highd 10y ago
Sorry you're getting down-voted, I don't think it's an unreasonable question. In the sense I think you're using it, "statistics" are really methods for dimensionality reduction - we take means, and medians and standard deviations with the hopes that they will capture the parts of the data we care about. This is important for two reasons - for one, for anything even moderately high dimension we'll never have enough data to be able to forego some means of aggregation due to the "curse of dimensionality". Secondly, the human-machine interaction information bandwidth is annoyingly low, so we need some way to compress any information for human consumption. "Statistics" are one way we do so. "Statistics" is also a field of study based around understanding how multiple data points relate to each other - that is of course critical to machine learning, and I think the terminology collision is why you're getting downvoted.

by bitL 10y ago

Normal_gaussian 10y ago

ML + Big Data are a specific application of statistics To to do anything beyond use tools other people have made (and never be sure whether results are meaningful or not) statistics are required Of course, to make money from the ML boom you can probably get away with coincidence and correlation

bitL 10y ago
Statistics means aggregate stuff and uses simplified characteristics out of semi-structured data. ML + Big Data allows you to ask precise questions like Where? How? Which ones?
- nfoz 10y ago
  As user "highd" suggested, I think you are confusing two words. I refer to Wiktionary for definitions: Statistics: A mathematical science concerned with data collection, presentation, analysis, and interpretation. Statistic: A quantity calculated from the data in a sample, which characterises an important aspect in the sample (such as mean or standard deviation). If "statistics" is the term for taking a mathematical approach to understanding data, then "machine learning" is basically an applied subset of that. But you seem to be specifically using the "a statistic" definition to describe what you think the "study of statistics" is entirely concerned with.

_Wintermute 10y ago

Yes we still need statistics. There is a huge overlap between machine learning methods and applied statistics, so much so that often there is not a clear distinction between the two.

j1vms 10y ago
> between machine learning methods and applied statistics (...) often there is not a clear distinction between the two. I would say applied statistics draws a line just prior to implementation concerns (say, real-world resource usage measured in time, space and energy) whereas these would be fully within scope and of interest in machine learning. As an example, applied statistics could provide a useful approach to a vision/image recognition problem, and this approach might be provably unrealizable in practice using real-world execution units (e.g. CUDA cores). Nonetheless, it might still be a very worthwhile theoretical result in applied statistics, although of no immediate interest within ML except to hint at potential new area of research.

joeyo 10y ago

It may not be true for all branches of machine learning (fuzzy logic, for example?), but the vast majority of modern ML techniques are equivalent to or can be viewed as types of statistical machine learning.

Normal_gaussian 10y ago
Good point about the fuzzy logic, often the boundaries between it and statistics are... fuzzy

klodolph 10y ago

Machine learning is often considered a statistical technique. The main difference seems to be that in traditional statistics, people derive practice from theory, whereas in ML people will try out techniques and figure out the theory later. That's really just a cultural difference. The techniques for analyzing ML models are all statistical to begin with. Statistics, as a field, already used general-purpose optimization algorithms before modern ML techniques came about, so in that sense, ML just fits into an existing position in the statistical toolbox (like replacing a chisel with a 3D printer). In the other direction, statistical techniques like cross-validation are necessary for you to get your ML correct.

bitL 10y ago
There is much more in ML than just statistics. I was basically asking why the "statistics filter" is so often on in ML. Neural networks don't seem a statistical technique, even if somebody uses them for regression. Yes, there is an overlap, but no, ML != statistics. As you mentioned, non-linear optimization is used in statistics on meta-level however nobody claims statistics is operations research or vice versa.
- 443633456 10y ago
  I agree with this comment because ML comprises some techniques that are still not understood [1]. I would like to think that statistics comes more from a pure math approach, loosely, while ML comes from an applied math approach, loosely. ML works spectacularly well on a class of problems. Why it does what it does is (in)conveniently brushed under the rug. How you treat that (in)convenience is left to you. [1] - https://www.quantamagazine.org/20151203-big-datas-mathematical-mysteries/ https://www.quantamagazine.org/20151203-big-datas-mathematic...
- srean 10y ago
  In what way is neural network not a statistical technique ? very curious to hear about your point of view. Statistics is not linear regression and ANOVA, or whatever catalogue of techniques in a freshman book, not even the library of techniques available in R. Statistics is the application of probability (or more broadly, math) to data. That said statisticians did miss the neural net wave because of their flippant reaction to it. They said, "oh well yet another non-parametric function approximator we have worked out the asymptotics 30 years ago". To paraphrase someone wise: asymptotically we are all dead. Not enough heed was paid to that. Among their other lacks were expertise in algorithms and optimization. Mind you, optimization has been at the core of their craft from their very genesis, its just that they did not feel it important enough to ride the cutting edge of research on optimization. Note: you cannot do maximum likelihood with solving an optimization problem. Gauss was doing it several hundreds of years ago for statistics. If I go on a bit further with my rant, they got a bit carried away with their fetish over bias and asymptotic normality. They missed the wave, sure. But all said and done, by any accepted definition of statistics, NN is very much also statistics.

highd 10y ago

Sorry you're getting down-voted, I don't think it's an unreasonable question. In the sense I think you're using it, "statistics" are really methods for dimensionality reduction - we take means, and medians and standard deviations with the hopes that they will capture the parts of the data we care about. This is important for two reasons - for one, for anything even moderately high dimension we'll never have enough data to be able to forego some means of aggregation due to the "curse of dimensionality". Secondly, the human-machine interaction information bandwidth is annoyingly low, so we need some way to compress any information for human consumption. "Statistics" are one way we do so. "Statistics" is also a field of study based around understanding how multiple data points relate to each other - that is of course critical to machine learning, and I think the terminology collision is why you're getting downvoted.