3 ms·

GPT-2 handles natural language which is specifically a time-series set, a sequence of natural word tokens. It is especially relevant because success in NLP task

GPT-2 handles natural language which is specifically a time-series set, a sequence of natural word tokens. It is especially relevant because success in NLP tasks requires tracking and learning long-term relationships - not just between words in a phrase or two, but between words and phrases in different paragraphs, in different parts of the text, so that a general coherence of the whole document is kept. This exactly what is done with long-term relationships.

curiousgal 7y ago
I know that NLP sequences are analogous to time series but GPT-2 can't be applied to say financial time series for example. That's what I meant.
- huffmsa 7y ago
  Or predicting the performance of a sports team over time. From what I've seen, It breaks down at the embedding layer, because while the teams "remain the same" in name / dictionary, their actual relative relationship to each other varies season to season / week to week.
  - NegatioN 7y ago
    My response here is purely intuition, since I have never worked much with time series. But wouldn't capturing that relationship require periodic retraining or other components to the network regardless? It may suggest that end-to-end training of a transformer is not suitable for these tasks, but that it might still capture the prediction of the long-scale time-series, if provided with extra data at each timestep in addition to the embeddings?
    - huffmsa 7y ago
      It does generate a long-term representation, the issue being that even with context and timestep specific data, that embedding is too general to make a good representation. Sports are particularly problematic because almost all teams and statistics regress to the average at some point, meaning your generated future timestep context clues don't really help modify the embedding. You're also dealing with variation within a season (injuries, better play, etc) and between seasons (personnel changes, rule changes, new stadiums, etc). So a team might have 4 seasons of above average performance, and then abruptly be the worst team in the league the next because they lost their coaches and star players.
fhars 7y ago
Then where is the publication that uses GPT-2 for a long-term NLP task, like, for example, reconstructing the rules of the great vowel shift from a corpus of all pre 20th century English? Synchronously parsing the meaning a single text has at one moment in time involves no time series at all.
- huffmsa 7y ago
  The parent comment isn't thinking time-series like you and I are. Being able to follow multiple agents and correctly deduce their relationships at a given time t is very hard. NLP "time-series" does a fine job at making back references within a text, but wouldn't be able to have multiple representations of a word or character through the years. It's very hard to get the computer to say "ah, the context is 16th century, so here are the relationships" without fudging it / tailoring models via tailored corpuses.
  - yorwba 7y ago
    Would adding a context vector for "16th century" be "fudging it"?
    - huffmsa 7y ago
      Depends. In the articles Uber example, the failure wasn't with detection, it as with context switching. The detection kept changing, and so the model kept going "oh, new object, restart decision process." Lacking the ability to generate and maintain it's own context is an area where a human would do better. We might not know what the object was, but our "slow down" response wouldn't keep resetting depending on what we classified the object as. Same as words switching meanings within a piece or sentence. It's hard, but most humans can pickup when the usage changes
- yorwba 7y ago
  How about a model that can detect the semantic shift of "Amazon" from "river" to "company"? https://arxiv.org/abs/1703.00607 https://arxiv.org/abs/1703.00607 They're not even using transformers, just simple word embeddings.

curiousgal 7y ago

I know that NLP sequences are analogous to time series but GPT-2 can't be applied to say financial time series for example. That's what I meant.

huffmsa 7y ago
Or predicting the performance of a sports team over time. From what I've seen, It breaks down at the embedding layer, because while the teams "remain the same" in name / dictionary, their actual relative relationship to each other varies season to season / week to week.
- NegatioN 7y ago
  My response here is purely intuition, since I have never worked much with time series. But wouldn't capturing that relationship require periodic retraining or other components to the network regardless? It may suggest that end-to-end training of a transformer is not suitable for these tasks, but that it might still capture the prediction of the long-scale time-series, if provided with extra data at each timestep in addition to the embeddings?
  - huffmsa 7y ago
    It does generate a long-term representation, the issue being that even with context and timestep specific data, that embedding is too general to make a good representation. Sports are particularly problematic because almost all teams and statistics regress to the average at some point, meaning your generated future timestep context clues don't really help modify the embedding. You're also dealing with variation within a season (injuries, better play, etc) and between seasons (personnel changes, rule changes, new stadiums, etc). So a team might have 4 seasons of above average performance, and then abruptly be the worst team in the league the next because they lost their coaches and star players.

fhars 7y ago

Then where is the publication that uses GPT-2 for a long-term NLP task, like, for example, reconstructing the rules of the great vowel shift from a corpus of all pre 20th century English? Synchronously parsing the meaning a single text has at one moment in time involves no time series at all.

huffmsa 7y ago
The parent comment isn't thinking time-series like you and I are. Being able to follow multiple agents and correctly deduce their relationships at a given time t is very hard. NLP "time-series" does a fine job at making back references within a text, but wouldn't be able to have multiple representations of a word or character through the years. It's very hard to get the computer to say "ah, the context is 16th century, so here are the relationships" without fudging it / tailoring models via tailored corpuses.
- yorwba 7y ago
  Would adding a context vector for "16th century" be "fudging it"?
  - huffmsa 7y ago
    Depends. In the articles Uber example, the failure wasn't with detection, it as with context switching. The detection kept changing, and so the model kept going "oh, new object, restart decision process." Lacking the ability to generate and maintain it's own context is an area where a human would do better. We might not know what the object was, but our "slow down" response wouldn't keep resetting depending on what we classified the object as. Same as words switching meanings within a piece or sentence. It's hard, but most humans can pickup when the usage changes
yorwba 7y ago
How about a model that can detect the semantic shift of "Amazon" from "river" to "company"? https://arxiv.org/abs/1703.00607 https://arxiv.org/abs/1703.00607 They're not even using transformers, just simple word embeddings.