4 ms·

(Author here) One of the things that confused me is that regression models can be predictive, just like time series forecasting — they just do so in a differen

by rdli 3y ago

(Author here) One of the things that confused me is that regression models can be predictive, just like time series forecasting — they just do so in a different way. I tried to make this clear in the article (or maybe I’m not understanding what you’re saying). In a regression model, you’re predicting target variables from feature variables. In a time series, you’re predicting the same variable from its past behavior. This is a subtle but crucial difference. (And then you can do time series with covariates, which combines the two.)

civilized 3y ago
Many of the most important time series prediction models are called "autoregressive", meaning they are regression models predicting the target from (prior values of) itself. This suggests that statisticians don't really share the view that these domains are distinct, or that regression models should only predict with different variables from the target.
- hackerlight 3y ago
  Right, AR(n) is a regression model, as are models which take only exogenous variables. My question is this. According to definitions, can the latter (f(X_t) = y_t) be a time series model if each row of data is a time step? It doesn't have any autoregressive terms in X, so I don't know if it categorically is a time-series model. Not that this question even matters, it's purely a taxonomy/terminology question.
  - nerdponx 3y ago
    Yes, it is. A time series model is any model where the data varies over time; that is, a time series model is any model of time series data. And timeseries data is broadly anything where the data for a single thing/entity varies over time. There are no strict definitions here, just common conventions.
    - hackerlight 3y ago
      Okay. And we can also say that there's some time series models that aren't regression models, right? For example, Kalman Filter is a "model" of a time series but isn't a regression.
      - nerdponx 3y ago
        Correct. Although the term "regression" is a misnomer anyway, and often when people say "regression" they mean "linear model". And by "linear model", we mean specifically a model in which outputs/predictions are some fixed linear combination of the input. It is however possible to interpret the Kalman filter as a kind of dynamic regression model. Check out here if you want a good math workout on that topic: https://stats.stackexchange.com/q/330696 https://stats.stackexchange.com/q/330696 (Another somewhat distinct meaning of the term "regression" is any model with a "continuous" outcome variable. This is usually in contrast to "classification", which is any model that has a "categorical" or discrete outcome variable.)
        hackerlight 3y ago
        I have a time series forecasting methodology question that I'll drop here. Suppose I have exogenous variables that vary over time, X(t). X is about 100 features. What are some methods I can apply onto X(t) to automatically engineer features that may be useful at predicting some noisy y(t)? I want to simultaneously capture interactions/interdependence between the columns of X, as well as the autocorrelation structure of X. If I treat X as merely tabular data, throwing it into a traditional regression model (e.g. XGBoost), it can capture the interdependence structure in X, but it will neglect the autocorrelation structure... Unless I manually engineer features that capture the autocorrelation structure in X (e.g. rolling/shifted/differenced features), but I want to explore methods that do that automatically.
        disgruntledphd2 3y ago
        if the variables in X(t) have the same time steps, I'd probably look at the cross correlation function of the X vs y, and then built another model on the X to predict X(t+n) and use that as an input for Y(t).
- nerdponx 3y ago
  Correct. In terms of what "kind" of model it is, it's all just a variation of the same linear model, y = bx. That said, there are a lot of special considerations involved with timeseries data. There is a large number of specialized tools, techniques, and model families dedicated to time series modeling, which don't make sense to use for other kinds of problems. And all of those special time series tools exist to solve problems that do not arise in other modeling situations. So in practice, times series modeling is a distinct specialization from other kinds of modeling.
taeric 3y ago
Apologies for dropping offline shortly after posting this. :( I should say that I enjoyed this post. And I think leaning into that confusion is my aim. In particular, my point of stochastic versus random is that they are more synonym than they are anything else. Just words that different groups came to use covering similar things. Which is not to say that their aren't differences in the crowd that uses each term. I posit that most of the differences is in the aims of the crowd, and at the end of the day, you can get a lot of mileage by embracing the similarities. As opposed to the default of contrasting on the differences. As a fun example, to me, if you view time not as just a number that always goes up, but as a number that cycles through the seasonal values, then it is easy to view as most any other feature. Similarly, the past is easy to envision as a feature of the present. I do think the way you described a lot of time series analysis fits the fun read I had where Mandlebrot proposed a fractal view of time series predictions. Where you are looking for self similar behavior in the series data and reflecting/overlaying it on itself. But... as is probably guessable from the rest of my post, a lot of this is far outside of my comfort area. Love reading about it from a distance.

by rdli 3y ago

civilized 3y ago

Many of the most important time series prediction models are called "autoregressive", meaning they are regression models predicting the target from (prior values of) itself. This suggests that statisticians don't really share the view that these domains are distinct, or that regression models should only predict with different variables from the target.

hackerlight 3y ago
Right, AR(n) is a regression model, as are models which take only exogenous variables. My question is this. According to definitions, can the latter (f(X_t) = y_t) be a time series model if each row of data is a time step? It doesn't have any autoregressive terms in X, so I don't know if it categorically is a time-series model. Not that this question even matters, it's purely a taxonomy/terminology question.
- nerdponx 3y ago
  Yes, it is. A time series model is any model where the data varies over time; that is, a time series model is any model of time series data. And timeseries data is broadly anything where the data for a single thing/entity varies over time. There are no strict definitions here, just common conventions.
  - hackerlight 3y ago
    Okay. And we can also say that there's some time series models that aren't regression models, right? For example, Kalman Filter is a "model" of a time series but isn't a regression.
    - nerdponx 3y ago
      Correct. Although the term "regression" is a misnomer anyway, and often when people say "regression" they mean "linear model". And by "linear model", we mean specifically a model in which outputs/predictions are some fixed linear combination of the input. It is however possible to interpret the Kalman filter as a kind of dynamic regression model. Check out here if you want a good math workout on that topic: https://stats.stackexchange.com/q/330696 https://stats.stackexchange.com/q/330696 (Another somewhat distinct meaning of the term "regression" is any model with a "continuous" outcome variable. This is usually in contrast to "classification", which is any model that has a "categorical" or discrete outcome variable.)
      - hackerlight 3y ago
        I have a time series forecasting methodology question that I'll drop here. Suppose I have exogenous variables that vary over time, X(t). X is about 100 features. What are some methods I can apply onto X(t) to automatically engineer features that may be useful at predicting some noisy y(t)? I want to simultaneously capture interactions/interdependence between the columns of X, as well as the autocorrelation structure of X. If I treat X as merely tabular data, throwing it into a traditional regression model (e.g. XGBoost), it can capture the interdependence structure in X, but it will neglect the autocorrelation structure... Unless I manually engineer features that capture the autocorrelation structure in X (e.g. rolling/shifted/differenced features), but I want to explore methods that do that automatically.
        disgruntledphd2 3y ago
        if the variables in X(t) have the same time steps, I'd probably look at the cross correlation function of the X vs y, and then built another model on the X to predict X(t+n) and use that as an input for Y(t).
nerdponx 3y ago
Correct. In terms of what "kind" of model it is, it's all just a variation of the same linear model, y = bx. That said, there are a lot of special considerations involved with timeseries data. There is a large number of specialized tools, techniques, and model families dedicated to time series modeling, which don't make sense to use for other kinds of problems. And all of those special time series tools exist to solve problems that do not arise in other modeling situations. So in practice, times series modeling is a distinct specialization from other kinds of modeling.

taeric 3y ago

Apologies for dropping offline shortly after posting this. :( I should say that I enjoyed this post. And I think leaning into that confusion is my aim. In particular, my point of stochastic versus random is that they are more synonym than they are anything else. Just words that different groups came to use covering similar things. Which is not to say that their aren't differences in the crowd that uses each term. I posit that most of the differences is in the aims of the crowd, and at the end of the day, you can get a lot of mileage by embracing the similarities. As opposed to the default of contrasting on the differences. As a fun example, to me, if you view time not as just a number that always goes up, but as a number that cycles through the seasonal values, then it is easy to view as most any other feature. Similarly, the past is easy to envision as a feature of the present. I do think the way you described a lot of time series analysis fits the fun read I had where Mandlebrot proposed a fractal view of time series predictions. Where you are looking for self similar behavior in the series data and reflecting/overlaying it on itself. But... as is probably guessable from the rest of my post, a lot of this is far outside of my comfort area. Love reading about it from a distance.