4 ms·

My response here is purely intuition, since I have never worked much with time series. But wouldn't capturing that relationship require periodic retraining or

My response here is purely intuition, since I have never worked much with time series. But wouldn't capturing that relationship require periodic retraining or other components to the network regardless? It may suggest that end-to-end training of a transformer is not suitable for these tasks, but that it might still capture the prediction of the long-scale time-series, if provided with extra data at each timestep in addition to the embeddings?

huffmsa 7y ago
It does generate a long-term representation, the issue being that even with context and timestep specific data, that embedding is too general to make a good representation. Sports are particularly problematic because almost all teams and statistics regress to the average at some point, meaning your generated future timestep context clues don't really help modify the embedding. You're also dealing with variation within a season (injuries, better play, etc) and between seasons (personnel changes, rule changes, new stadiums, etc). So a team might have 4 seasons of above average performance, and then abruptly be the worst team in the league the next because they lost their coaches and star players.