4 ms·

This article seems out-of-date by 5 years or more even though it was published today, and I am unsure as to why. It calls out long short term memory but doesn'

This article seems out-of-date by 5 years or more even though it was published today, and I am unsure as to why. It calls out long short term memory but doesn't mention recent (last 5 years) improvements like Gated Recurrent Networks (GRUs) or Transformers (GPT-2, huggingface/transformers) which have shown significant improvements over the traditional LSTM model. These can handle time series data much better than older models could.

joe_the_user 7y ago
I think you need to give details and references to support a claim that these innovations make a fundamental difference. I don't doubt that the things you mention involve improvements but are these improvements doing better on the same benchmarks in the same fashion or a fundamental change. I read many claims that recent changes in deep learning represent the former.
- skunkworker 7y ago
  It's to the point now where using a basic LSTM is almost discouraged compared to using a Transformer. GPT-2 wouldn't have been possible without these recent innovations. Here are some good guides on Transformers [1] and attention/ multi headed attention [2], as well as the paper that proposed the transformer model "Attention is all you need" [3]. GPT-2 heavily relies upon the advancements that transformers brought [4] [1] http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/ [2] https://towardsdatascience.com/attention-for-time-series-classification-and-forecasting-261723e0006d https://towardsdatascience.com/attention-for-time-series-cla... [3] Attention is all you need. https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762 [4] http://jalammar.github.io/illustrated-gpt2/ http://jalammar.github.io/illustrated-gpt2/
  - curiousgal 7y ago
    Except GPT-2 is irrelevant when it comes to time series specifically.
    - Erlich_Bachman 7y ago
      GPT-2 handles natural language which is specifically a time-series set, a sequence of natural word tokens. It is especially relevant because success in NLP tasks requires tracking and learning long-term relationships - not just between words in a phrase or two, but between words and phrases in different paragraphs, in different parts of the text, so that a general coherence of the whole document is kept. This exactly what is done with long-term relationships.
      - curiousgal 7y ago
        I know that NLP sequences are analogous to time series but GPT-2 can't be applied to say financial time series for example. That's what I meant.
        huffmsa 7y ago
        Or predicting the performance of a sports team over time. From what I've seen, It breaks down at the embedding layer, because while the teams "remain the same" in name / dictionary, their actual relative relationship to each other varies season to season / week to week.
        NegatioN 7y ago
        My response here is purely intuition, since I have never worked much with time series. But wouldn't capturing that relationship require periodic retraining or other components to the network regardless? It may suggest that end-to-end training of a transformer is not suitable for these tasks, but that it might still capture the prediction of the long-scale time-series, if provided with extra data at each timestep in addition to the embeddings?
        huffmsa 7y ago
        It does generate a long-term representation, the issue being that even with context and timestep specific data, that embedding is too general to make a good representation. Sports are particularly problematic because almost all teams and statistics regress to the average at some point, meaning your generated future timestep context clues don't really help modify the embedding. You're also dealing with variation within a season (injuries, better play, etc) and between seasons (personnel changes, rule changes, new stadiums, etc). So a team might have 4 seasons of above average performance, and then abruptly be the worst team in the league the next because they lost their coaches and star players.
  - joe_the_user 7y ago
    It's to the point now where using a basic LSTM is almost discouraged compared to using a Transformer. GPT-2 wouldn't have been possible without these recent innovations. So, but that "point" could just be that being a published academic means always using the latest thing. That doesn't demonstrate how much better these are, much less demonstrate that they represent fundamental steps forward.
  - siekmanj 7y ago
    This is mostly true for supervised and unsupervised learning models, but for reinforcement learning the LSTM is king because of the convenient fact that it can be evaluated one time step at a time, instead of just outputting a sequence like a transformer. For things like robotic control, etc, attention-based models are pretty nonsensical.
    - jeremysalwen 7y ago
      Not true, a transformer can be used in models without any lookahead, for example how it is used in gpt-2.! The real difference is the complexity of the model and the large increase in computational cost.
- laingc 7y ago
  I have no issue with you asking for a reference for this, but when I read your comment I did a double-take. To someone familiar with deep learning, this claim is completely self-evident. The article is indeed almost farcically out of date, and LSTMs haven’t been close to the state of the art for years now.
hnews_account_1 7y ago
Any links to heavy time series based machine learning algorithms? I'm in finance, and while I know how to establish and run a random forest or gradient boost regressor using standard libraries, I've never had a good handle on them.
- deleted 7y ago
  [deleted]
- cbsmith 7y ago
  Most everything Eamonn Keogh publishes: https://www.cs.ucr.edu/%7Eeamonn/selected_publications.htm https://www.cs.ucr.edu/%7Eeamonn/selected_publications.htm
- throwawaymath 7y ago
  Look into matrix profiles and associated algorithms.
lern_too_spel 7y ago
Even weirder, it calls LSTM a "newly invented variant" of RNN. LSTM is 20 years old.

joe_the_user 7y ago

I think you need to give details and references to support a claim that these innovations make a fundamental difference. I don't doubt that the things you mention involve improvements but are these improvements doing better on the same benchmarks in the same fashion or a fundamental change. I read many claims that recent changes in deep learning represent the former.

skunkworker 7y ago
It's to the point now where using a basic LSTM is almost discouraged compared to using a Transformer. GPT-2 wouldn't have been possible without these recent innovations. Here are some good guides on Transformers [1] and attention/ multi headed attention [2], as well as the paper that proposed the transformer model "Attention is all you need" [3]. GPT-2 heavily relies upon the advancements that transformers brought [4] [1] http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/ [2] https://towardsdatascience.com/attention-for-time-series-classification-and-forecasting-261723e0006d https://towardsdatascience.com/attention-for-time-series-cla... [3] Attention is all you need. https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762 [4] http://jalammar.github.io/illustrated-gpt2/ http://jalammar.github.io/illustrated-gpt2/
- curiousgal 7y ago
  Except GPT-2 is irrelevant when it comes to time series specifically.
  - Erlich_Bachman 7y ago
    GPT-2 handles natural language which is specifically a time-series set, a sequence of natural word tokens. It is especially relevant because success in NLP tasks requires tracking and learning long-term relationships - not just between words in a phrase or two, but between words and phrases in different paragraphs, in different parts of the text, so that a general coherence of the whole document is kept. This exactly what is done with long-term relationships.
    - curiousgal 7y ago
      I know that NLP sequences are analogous to time series but GPT-2 can't be applied to say financial time series for example. That's what I meant.
      - huffmsa 7y ago
        Or predicting the performance of a sports team over time. From what I've seen, It breaks down at the embedding layer, because while the teams "remain the same" in name / dictionary, their actual relative relationship to each other varies season to season / week to week.
        NegatioN 7y ago
        My response here is purely intuition, since I have never worked much with time series. But wouldn't capturing that relationship require periodic retraining or other components to the network regardless? It may suggest that end-to-end training of a transformer is not suitable for these tasks, but that it might still capture the prediction of the long-scale time-series, if provided with extra data at each timestep in addition to the embeddings?
        huffmsa 7y ago
        It does generate a long-term representation, the issue being that even with context and timestep specific data, that embedding is too general to make a good representation. Sports are particularly problematic because almost all teams and statistics regress to the average at some point, meaning your generated future timestep context clues don't really help modify the embedding. You're also dealing with variation within a season (injuries, better play, etc) and between seasons (personnel changes, rule changes, new stadiums, etc). So a team might have 4 seasons of above average performance, and then abruptly be the worst team in the league the next because they lost their coaches and star players.
- joe_the_user 7y ago
  It's to the point now where using a basic LSTM is almost discouraged compared to using a Transformer. GPT-2 wouldn't have been possible without these recent innovations. So, but that "point" could just be that being a published academic means always using the latest thing. That doesn't demonstrate how much better these are, much less demonstrate that they represent fundamental steps forward.
- siekmanj 7y ago
  This is mostly true for supervised and unsupervised learning models, but for reinforcement learning the LSTM is king because of the convenient fact that it can be evaluated one time step at a time, instead of just outputting a sequence like a transformer. For things like robotic control, etc, attention-based models are pretty nonsensical.
  - jeremysalwen 7y ago
    Not true, a transformer can be used in models without any lookahead, for example how it is used in gpt-2.! The real difference is the complexity of the model and the large increase in computational cost.
laingc 7y ago
I have no issue with you asking for a reference for this, but when I read your comment I did a double-take. To someone familiar with deep learning, this claim is completely self-evident. The article is indeed almost farcically out of date, and LSTMs haven’t been close to the state of the art for years now.

hnews_account_1 7y ago

Any links to heavy time series based machine learning algorithms? I'm in finance, and while I know how to establish and run a random forest or gradient boost regressor using standard libraries, I've never had a good handle on them.

deleted 7y ago
[deleted]
cbsmith 7y ago
Most everything Eamonn Keogh publishes: https://www.cs.ucr.edu/%7Eeamonn/selected_publications.htm https://www.cs.ucr.edu/%7Eeamonn/selected_publications.htm
throwawaymath 7y ago
Look into matrix profiles and associated algorithms.

lern_too_spel 7y ago

Even weirder, it calls LSTM a "newly invented variant" of RNN. LSTM is 20 years old.