4 ms·

As usual in ML, the appropriate solution depends on the problem and context. ML (particularly DL) tends to outperform "classical" statistical time series forec

As usual in ML, the appropriate solution depends on the problem and context. ML (particularly DL) tends to outperform "classical" statistical time series forecasting when the data is (strongly) nonlinear, highly dimensional and large. The opposite holds as well. It is also important to note that accuracy is not the only relevant metric in practical applications. Explainability is of particular interest in time series forecasting: it is good to know if your sales are going to increase/decrease, but it is even more valuable to know which input variables are likely to account for that change. Hence, a "simple" model with inferior forecasting accuracy might be preferred to a stronger estimator if it can give insights to not only the "what" will happen, but also the "why".

tomwphillips 4y ago
> ML (particularly DL) tends to outperform "classical" statistical time series forecasting when the data is (strongly) nonlinear, highly dimensional and large. This claim about forecasting with DL comes up a lot, but I’ve seen little evidence to back it up. Personally, I’ve never managed to have the same success others apparently have with DL time series forecasting.
- beernet 4y ago
  It's true simply because large ANNs have a higher capacity, which is great for large, nonlinear data but less so for small datasets or simple functions. In any case, Transformers are eating ML right now and I'm actually surprised there's no "GPT-3 for time series" yet. It's technically the same problem as language modeling (that is, multi-step prediction of numerics), however, there is only a comparably little amount of human-generated data for self-supervised learning of a time series forecasting model. Another reason might be that the expected applications and potentials of such a pre-trained model aren't as glamorous as generating language.
  - time_to_smile 4y ago
    > It's technically the same problem as language modeling You're thinking of modeling event sequences which is not strictly speaking the same as time series modeling. Plenty of people do use LSTMs to model event sequences, using the hidden state of the model as a vector representation of processes current location walking a graph (i.e. a Users journey through a mobile app, or navigating following links on the web.) Time series is different because the ticks of timed events are at consistent intervals and are also part of the problem being modeled. In general time series models have often been distinct from sequence models. The reason there's no GPT-3 for any general sequence is the lack of data. Typically the vocabulary of events is much smaller than natural languages and the corpus of sequences much smaller.
    - jmmcd 4y ago
      There's a deeper issue. All language (and code and other things in the GPT/etc corpora) seem to have something in common - hierarchical, short- and long-range structure. In contrast, there is nothing that all time series have in common. There's no way to learn generic time series knowledge that will reliably generalise to new unseen time series.
  - tomwphillips 4y ago
    Like I said, still not seen any evidence.
    - beernet 4y ago
      Then look at some of the past time series related Kaggle challenges, plenty of evidence there in the winning solutions.
time_to_smile 4y ago
A larger problem is that time series modeling is particularly resistant towards black box approaches since a lot of information is encoded in the model itself. Take even a simple moving average model on daily observations. Consider stock ticker data (where there are no weekends) and web traffic data (where there is an observation each day). The stock ticker data should be smoothed with a 5 day window and the web traffic with a 7 to help reduce the impact of weekly effects (which probably shouldn't exist in the stock market anyway). It's possible in either of these cases you might find a moving average that performs better on some choose metric, say 4 or 8 days. However neither of these alternatives make any sense as a window if we're trying to remove day-of-week effect, and unless you can come up with a justifiable explanation, smoothing over arbitrary windows should be avoided. If you let a black box optimize even a simple moving average you would be avoiding some very essential introspection into what your model is actually claiming. Not to mention that we often can do more than just prediction with these intentional model tunings (for example day-of-week effect can be explicitly differenced from the data to measure exactly how much sales should increase on a Saturday)
- throwawaythekey 4y ago
  Do you know of any good resources that explain why smoothing should be used? My intuition is that in your given example (stock prices) smoothing would probably be doing yourself a disservice as it would hide the optimal hour or day of the week to make purchases/sales. Is it mostly related to the timeframe of your analysis and needing to trade off near term precision for longer term precision?
  - time_to_smile 4y ago
    Smoothing is used to negate seasonal effects. Suppose you run a bar, and your busiest days in order are Friday, Saturday, Thursday, Sunday, Wednesday, Tuesday and Monday. Now you are the owner and you want to look at your foot traffic everyday to monitor the health of your business. However, from the ordering I've presented, this will almost never work trivially. Monday traffic will always be less than Sunday, does this mean every Monday you should be concerned about business? Of course not. However by averaging the last 7 days and looking at that each day, you are canceling out these seasonal effects because every single day of the week is accounted for in your measurement. If the 7 day moving average on Monday is less than Sunday you should be concerned because the average when calculated on Sunday included the pervious Monday. For your example, you use the smoothed data and a history of the original data to come up with an exact explanation of which days are the best. For example if you are a bar owner and you don't know which day is the best, you can take a 7 day moving average and subtract it from each day of actual observations. Then averages those differences grouped by day of week and you get an estimate for the day of week effect (you can also calculate standard deviation as well).
RA_Fisher 4y ago
So true, causal inference is much more valuable than prediction which is a relatively cheap commodity.

tomwphillips 4y ago

> ML (particularly DL) tends to outperform "classical" statistical time series forecasting when the data is (strongly) nonlinear, highly dimensional and large. This claim about forecasting with DL comes up a lot, but I’ve seen little evidence to back it up. Personally, I’ve never managed to have the same success others apparently have with DL time series forecasting.

beernet 4y ago
It's true simply because large ANNs have a higher capacity, which is great for large, nonlinear data but less so for small datasets or simple functions. In any case, Transformers are eating ML right now and I'm actually surprised there's no "GPT-3 for time series" yet. It's technically the same problem as language modeling (that is, multi-step prediction of numerics), however, there is only a comparably little amount of human-generated data for self-supervised learning of a time series forecasting model. Another reason might be that the expected applications and potentials of such a pre-trained model aren't as glamorous as generating language.
- time_to_smile 4y ago
  > It's technically the same problem as language modeling You're thinking of modeling event sequences which is not strictly speaking the same as time series modeling. Plenty of people do use LSTMs to model event sequences, using the hidden state of the model as a vector representation of processes current location walking a graph (i.e. a Users journey through a mobile app, or navigating following links on the web.) Time series is different because the ticks of timed events are at consistent intervals and are also part of the problem being modeled. In general time series models have often been distinct from sequence models. The reason there's no GPT-3 for any general sequence is the lack of data. Typically the vocabulary of events is much smaller than natural languages and the corpus of sequences much smaller.
  - jmmcd 4y ago
    There's a deeper issue. All language (and code and other things in the GPT/etc corpora) seem to have something in common - hierarchical, short- and long-range structure. In contrast, there is nothing that all time series have in common. There's no way to learn generic time series knowledge that will reliably generalise to new unseen time series.
- tomwphillips 4y ago
  Like I said, still not seen any evidence.
  - beernet 4y ago
    Then look at some of the past time series related Kaggle challenges, plenty of evidence there in the winning solutions.

time_to_smile 4y ago

A larger problem is that time series modeling is particularly resistant towards black box approaches since a lot of information is encoded in the model itself. Take even a simple moving average model on daily observations. Consider stock ticker data (where there are no weekends) and web traffic data (where there is an observation each day). The stock ticker data should be smoothed with a 5 day window and the web traffic with a 7 to help reduce the impact of weekly effects (which probably shouldn't exist in the stock market anyway). It's possible in either of these cases you might find a moving average that performs better on some choose metric, say 4 or 8 days. However neither of these alternatives make any sense as a window if we're trying to remove day-of-week effect, and unless you can come up with a justifiable explanation, smoothing over arbitrary windows should be avoided. If you let a black box optimize even a simple moving average you would be avoiding some very essential introspection into what your model is actually claiming. Not to mention that we often can do more than just prediction with these intentional model tunings (for example day-of-week effect can be explicitly differenced from the data to measure exactly how much sales should increase on a Saturday)

throwawaythekey 4y ago
Do you know of any good resources that explain why smoothing should be used? My intuition is that in your given example (stock prices) smoothing would probably be doing yourself a disservice as it would hide the optimal hour or day of the week to make purchases/sales. Is it mostly related to the timeframe of your analysis and needing to trade off near term precision for longer term precision?
- time_to_smile 4y ago
  Smoothing is used to negate seasonal effects. Suppose you run a bar, and your busiest days in order are Friday, Saturday, Thursday, Sunday, Wednesday, Tuesday and Monday. Now you are the owner and you want to look at your foot traffic everyday to monitor the health of your business. However, from the ordering I've presented, this will almost never work trivially. Monday traffic will always be less than Sunday, does this mean every Monday you should be concerned about business? Of course not. However by averaging the last 7 days and looking at that each day, you are canceling out these seasonal effects because every single day of the week is accounted for in your measurement. If the 7 day moving average on Monday is less than Sunday you should be concerned because the average when calculated on Sunday included the pervious Monday. For your example, you use the smoothed data and a history of the original data to come up with an exact explanation of which days are the best. For example if you are a bar owner and you don't know which day is the best, you can take a 7 day moving average and subtract it from each day of actual observations. Then averages those differences grouped by day of week and you get an estimate for the day of week effect (you can also calculate standard deviation as well).

RA_Fisher 4y ago

So true, causal inference is much more valuable than prediction which is a relatively cheap commodity.