6 ms·

Deep learning machine learning models are statistical and probabilistic models. You can categorize Deep Learning under both computer science and statistics. F

Deep learning machine learning models are statistical and probabilistic models. You can categorize Deep Learning under both computer science and statistics. For example stat.ML and cs.LG in Arxiv. Machine learning and statistics are closely related fields, both historically and in current practice and methodology.

sgt101 5y ago
My working definition is that statisticians choose and engineer models while machine learning searches a vast space of models.
- nightski 5y ago
  That doesn't seem right to me. In both cases you have a model and are just searching for optimal parameters considering the bias/variance tradeoff. There may be a few instances of a neural network or other ML model being set up to dynamically change it's architecture during training but that seems to me like it would not work out well at all. If anything in specific cases the statistical model (if Bayesian) is more comprehensive in that it doesn't try to find a point estimate of the parameters but instead forms a full distribution around the plausibility of the parameters.
  - sjg007 5y ago
    You can have Bayesian DNNs.
    - whimsicalism 5y ago
      Not easily. I don't think the literature is very incredible on this - how do you define a prior over all of the parameters of an NN?
      - sjg007 5y ago
        There’s a vast literature you can read.
        whimsicalism 5y ago
        I have read some of the literature - ie. the bayes by backprop method, etc. It doesn't seem like getting a posterior over the parameter space of a neural network is tractable as of now.
        sjg007 5y ago
        It ain't tractable in general either.
  - sgt101 5y ago
    So I was thinking like this - if you have a data set with 200000 variables and 70,000 examples how would you find a model without using a search process? On the other hand if you have 20 variables and or if you understand which of the 200000 variables are the ones to worry about then you can build a model by hand. I guess that also statisticians are working to summarise or create insight about data while ML is working to create a prediction (although statistical models can be used to do that too).
    - bart_spoon 5y ago
      Why do you think that statisticians only build models by hand? Dimension reduction techniques are employed in statistical learning as well.
- sdenton4 5y ago
  Another way I've heard it framed is that statistics cares about the parameters, and ML cares about the answers.
  - fighterpilot 5y ago
    ML cares about predictive generalization, stats cares about understanding and interpreting drivers
  - gmfawcett 5y ago
    That framing seems a wee bit dismissive of statistics, esp. applied stats -- I'll hazard a guess that this came from the ML camp. :)
    - srean 5y ago
      There is nothing dismissive about it, they are just different things. If we have strong theoretical understanding of the physics/model of a problem, but are unsure about some parameters, it makes sense to develop methods to find those unknown parameters accurately. This ability is a big deal in traditional statistics. People working in this field try and prove results that their proposed method can actually do this. If the model happens to be largely correct and the method robust, this even allows prediction. Often, however, we do not know the model and the 'parameter' is a piece of fiction anyway. If we are interested in prediction alone, its fine to let go of an ability to accurately estimate the parameters as long as predictions are accurate. Think epicycle models of planetary motion. ML folks try to prove that their methods have good prediction properties and are happy to sacrifice on parameter recovery. Nonparametric statistics and prequential statistics are somewhere in the middle. Sometimes it does seem though that people are trying to out do each other, coming up with methods that estimates the spectrum of a unicorn's rainbow more accurately than the best known result in research literature. This may look odd because the unicorn and his rainbow are pieces of fiction.
      - gmfawcett 5y ago
        Fair points made, and your unicorn analogy is wonderful. :)

sgt101 5y ago

My working definition is that statisticians choose and engineer models while machine learning searches a vast space of models.

nightski 5y ago
That doesn't seem right to me. In both cases you have a model and are just searching for optimal parameters considering the bias/variance tradeoff. There may be a few instances of a neural network or other ML model being set up to dynamically change it's architecture during training but that seems to me like it would not work out well at all. If anything in specific cases the statistical model (if Bayesian) is more comprehensive in that it doesn't try to find a point estimate of the parameters but instead forms a full distribution around the plausibility of the parameters.
- sjg007 5y ago
  You can have Bayesian DNNs.
  - whimsicalism 5y ago
    Not easily. I don't think the literature is very incredible on this - how do you define a prior over all of the parameters of an NN?
    - sjg007 5y ago
      There’s a vast literature you can read.
      - whimsicalism 5y ago
        I have read some of the literature - ie. the bayes by backprop method, etc. It doesn't seem like getting a posterior over the parameter space of a neural network is tractable as of now.
        sjg007 5y ago
        It ain't tractable in general either.
- sgt101 5y ago
  So I was thinking like this - if you have a data set with 200000 variables and 70,000 examples how would you find a model without using a search process? On the other hand if you have 20 variables and or if you understand which of the 200000 variables are the ones to worry about then you can build a model by hand. I guess that also statisticians are working to summarise or create insight about data while ML is working to create a prediction (although statistical models can be used to do that too).
  - bart_spoon 5y ago
    Why do you think that statisticians only build models by hand? Dimension reduction techniques are employed in statistical learning as well.
sdenton4 5y ago
Another way I've heard it framed is that statistics cares about the parameters, and ML cares about the answers.
- fighterpilot 5y ago
  ML cares about predictive generalization, stats cares about understanding and interpreting drivers
- gmfawcett 5y ago
  That framing seems a wee bit dismissive of statistics, esp. applied stats -- I'll hazard a guess that this came from the ML camp. :)
  - srean 5y ago
    There is nothing dismissive about it, they are just different things. If we have strong theoretical understanding of the physics/model of a problem, but are unsure about some parameters, it makes sense to develop methods to find those unknown parameters accurately. This ability is a big deal in traditional statistics. People working in this field try and prove results that their proposed method can actually do this. If the model happens to be largely correct and the method robust, this even allows prediction. Often, however, we do not know the model and the 'parameter' is a piece of fiction anyway. If we are interested in prediction alone, its fine to let go of an ability to accurately estimate the parameters as long as predictions are accurate. Think epicycle models of planetary motion. ML folks try to prove that their methods have good prediction properties and are happy to sacrifice on parameter recovery. Nonparametric statistics and prequential statistics are somewhere in the middle. Sometimes it does seem though that people are trying to out do each other, coming up with methods that estimates the spectrum of a unicorn's rainbow more accurately than the best known result in research literature. This may look odd because the unicorn and his rainbow are pieces of fiction.
    - gmfawcett 5y ago
      Fair points made, and your unicorn analogy is wonderful. :)