5 ms·

Nice article and interesting comparison. Yet, I have a minor issue with the title: Deep Learning are also statistical methods ... "univariate models vs. " would

Nice article and interesting comparison. Yet, I have a minor issue with the title: Deep Learning are also statistical methods ... "univariate models vs. " would be a better title.

nerdponx 4y ago
You could argue that deep learning is not a statistical method in the traditional sense, in that a typical neural network model is not a probability model, and some neural networks are well known to produce specifically bad probability models, requiring some amount of post processing in order to produce correctly "calibrated" probability predictions. However I don't like that there is often a strict dichotomy presented between "deep learning" and "statistics". There is a whole world of gray areas and hybrid techniques, which tend to be both more accessible, easier to reason about, and more effective in practice, especially on smaller "tabular" datasets. What about generalized additive models, random forests, gradient boosted trees, etc.? The author of the document I'm sure is aware of these techniques, and I assume they are left out because they didn't perform well enough to be considered here. But I don't think it does the discourse any favors to promulgate the false dichotomy.
- fedegr 4y ago
  Co-author here: all in due time. Next iteration we will include LigthGBM, XGBoost, and newer DL models like TFT and NHiTS.
- uoaei 4y ago
  Statistical models and probabilistic models are not synonymous. Vanilla deep learning models are statistical models (a la linear regression) and not probabilistic models (a la Gaussian mixture). It is important to maintain the distinction. But to your point about the dichotomy between deep learning and more "traditional" statistical methods: this confusion in common parlance clearly has negative effects on model-building among engineers. You are right that when people think "deep learning" they think of very specific architectures with very specific features, and don't seem to conceive of the possibility that automatic differentiation techniques mean you can incorporate all sorts of new model components that blur the line between deep learning and older methods. For instance, you could feed the results of a kernel SVM to an ARIMA model in such a way that the whole thing is end-to-end differentiable. In fact, the great benefit of deep learning long-term is (in my opinion) that the ability to build these compositional models means you can bake in that much more inductive bias into the models you build, meaning they can be smaller and more stable in training.
  - dumb1224 4y ago
    I have very limited statistical background but doesn't variational inference applied in the neural networks make them probabilistic models? The modelling definitely seems so because the math in those papers doesn't even specify whether it's a network (it implies that it can be any model).
    - uoaei 4y ago
      Yes indeed. This synthesis of concepts is a great illustration of moving beyond hardened dichotomies in this research space and I believe similar approaches will be fruitful in the years to come.
  - salty_biscuits 4y ago
    "Vanilla deep learning models are statistical models (a la linear regression) and not probabilistic models (a la Gaussian mixture). It is important to maintain the distinction." Isn't this just a matter of interpretation of the models? You can interpret linear regression in a Bayesian way and say that the prediction of the linear model is the MAP of the mean, you can also calculate the variance, the l2 norm objective is saying the distribution of errors is normally distributed, l2 regularisation is a normal prior on the coefficients, etc, etc? All the same stuff can be applied to deep learning models. Maybe I don't understand your distinction between statistical and probabilistic though?
    - uoaei 4y ago
      > Isn't this just a matter of interpretation of the models? Not really. This is the classic frequentist vs Bayesian debate. In frequentist-land, you are computing point estimates of the model parameters. In Bayesian-land, you are computing distribution estimates of the model parameters. It is true that there is a difference in interpretation of the generative process but the two choices demand fundamentally different models because of the decision about which of the parameters or data are considered "real" and which are considered "generated". I think a more abstract/general way to put it is: "statistics" is concerned with statistical summary values (i.e. mean-field estimates over measures) while "probability" is concerned more with distributions (i.e., topologies of measures). I'm not sure this is a rigorously correct way to characterize it, but it illustrates the intuition I'm trying to convey.
stellalo 4y ago
They are all univariate models: some are trained offline on a bunch of different series before being applied (deep learning, “global” models), others are applied directly to each series to forecast (“statistical”, “local” models), but the task is the same univariate time series prediction for every model there.

Nice article and interesting comparison. Yet, I have a minor issue with the title: Deep Learning are also statistical methods ... "univariate models vs. " would be a better title.

nerdponx 4y ago

You could argue that deep learning is not a statistical method in the traditional sense, in that a typical neural network model is not a probability model, and some neural networks are well known to produce specifically bad probability models, requiring some amount of post processing in order to produce correctly "calibrated" probability predictions. However I don't like that there is often a strict dichotomy presented between "deep learning" and "statistics". There is a whole world of gray areas and hybrid techniques, which tend to be both more accessible, easier to reason about, and more effective in practice, especially on smaller "tabular" datasets. What about generalized additive models, random forests, gradient boosted trees, etc.? The author of the document I'm sure is aware of these techniques, and I assume they are left out because they didn't perform well enough to be considered here. But I don't think it does the discourse any favors to promulgate the false dichotomy.

fedegr 4y ago
Co-author here: all in due time. Next iteration we will include LigthGBM, XGBoost, and newer DL models like TFT and NHiTS.
uoaei 4y ago
Statistical models and probabilistic models are not synonymous. Vanilla deep learning models are statistical models (a la linear regression) and not probabilistic models (a la Gaussian mixture). It is important to maintain the distinction. But to your point about the dichotomy between deep learning and more "traditional" statistical methods: this confusion in common parlance clearly has negative effects on model-building among engineers. You are right that when people think "deep learning" they think of very specific architectures with very specific features, and don't seem to conceive of the possibility that automatic differentiation techniques mean you can incorporate all sorts of new model components that blur the line between deep learning and older methods. For instance, you could feed the results of a kernel SVM to an ARIMA model in such a way that the whole thing is end-to-end differentiable. In fact, the great benefit of deep learning long-term is (in my opinion) that the ability to build these compositional models means you can bake in that much more inductive bias into the models you build, meaning they can be smaller and more stable in training.
- dumb1224 4y ago
  I have very limited statistical background but doesn't variational inference applied in the neural networks make them probabilistic models? The modelling definitely seems so because the math in those papers doesn't even specify whether it's a network (it implies that it can be any model).
  - uoaei 4y ago
    Yes indeed. This synthesis of concepts is a great illustration of moving beyond hardened dichotomies in this research space and I believe similar approaches will be fruitful in the years to come.
- salty_biscuits 4y ago
  "Vanilla deep learning models are statistical models (a la linear regression) and not probabilistic models (a la Gaussian mixture). It is important to maintain the distinction." Isn't this just a matter of interpretation of the models? You can interpret linear regression in a Bayesian way and say that the prediction of the linear model is the MAP of the mean, you can also calculate the variance, the l2 norm objective is saying the distribution of errors is normally distributed, l2 regularisation is a normal prior on the coefficients, etc, etc? All the same stuff can be applied to deep learning models. Maybe I don't understand your distinction between statistical and probabilistic though?
  - uoaei 4y ago
    > Isn't this just a matter of interpretation of the models? Not really. This is the classic frequentist vs Bayesian debate. In frequentist-land, you are computing point estimates of the model parameters. In Bayesian-land, you are computing distribution estimates of the model parameters. It is true that there is a difference in interpretation of the generative process but the two choices demand fundamentally different models because of the decision about which of the parameters or data are considered "real" and which are considered "generated". I think a more abstract/general way to put it is: "statistics" is concerned with statistical summary values (i.e. mean-field estimates over measures) while "probability" is concerned more with distributions (i.e., topologies of measures). I'm not sure this is a rigorously correct way to characterize it, but it illustrates the intuition I'm trying to convey.

stellalo 4y ago

They are all univariate models: some are trained offline on a bunch of different series before being applied (deep learning, “global” models), others are applied directly to each series to forecast (“statistical”, “local” models), but the task is the same univariate time series prediction for every model there.