4 ms·

If I want to do sequence modelling, let's say, predict the 9th element of the following sequence: [1, 2, 3, 4, 5, 6, 7, 8, 9] -- that is, I know the 8 most rece

If I want to do sequence modelling, let's say, predict the 9th element of the following sequence: [1, 2, 3, 4, 5, 6, 7, 8, 9] -- that is, I know the 8 most recent tokens as my "context", and I want to predict the next, 9 in this case -- Can someone explain to me why a transformer or RNN is better at this than a simple linear layer with an equivalent number of parameters? A linear layer can receive the context [1, 2, 3, 4, 5, 6, 7, 8], properly one-hot encoded / embedded, etc, and predict the next sequence. Can a linear layer do just as well as a transformer? This setup allows linear layers to predict sequences with an arbitrary context size, so why so much hype about transformers and RNNs and other sequence focused architectures? Perhaps the difference is that given the same number of parameters, the transformer uses those parameters to perform easy computations whereas the linear layer just does one gigantic matrix multiplication which isn't very efficient?

QuadmasterXLII 4y ago
The big difference is that the transformer is (approximately) permutation equivariant, which makes a massive difference in generalization and training speed.
- Buttons840 4y ago
  I see, so [1, 2, 3, 4, 5, 6, 7, 8] is more similar to [5, 3, 1, 7, 8, 2, 4, 6] than with a linear layer? That's what you mean by permutation equivariant? I understand each context input is embedded with its position, but I suppose the transformer can learn to ignore the position and just look at the context as an unordered set?
- eachro 4y ago
  What makes it approximately permutation equivariant (vs entirely)? As I understand things, if the order is jumbled, the attention matrix does get its rows and cols permuted in the way you'd expect so I'd have thought they'd be entirely permutation equivariant.
  - YetAnotherNick 4y ago
    Inputs have position encoding in them.
spyder 4y ago
Yea, this paper says that the embarrassingly simple linear is better than transformers for time-series: https://arxiv.org/abs/2205.13504 https://arxiv.org/abs/2205.13504 from the abstract: "Recently, there has been a surge of Transformer-based solutions for the long-term time series forecasting (LTSF) task. Despite the growing performance over the past few years, we question the validity of this line of research in this work. Specifically, Transformers is arguably the most successful solution to extract the semantic correlations among the elements in a long sequence. However, in time series modeling, we are to extract the temporal relations in an ordered set of continuous points. While employing positional encoding and using tokens to embed sub-series in Transformers facilitate preserving some ordering information, the nature of the permutation-invariant self-attention mechanism inevitably results in temporal information loss..."

QuadmasterXLII 4y ago

The big difference is that the transformer is (approximately) permutation equivariant, which makes a massive difference in generalization and training speed.

Buttons840 4y ago
I see, so [1, 2, 3, 4, 5, 6, 7, 8] is more similar to [5, 3, 1, 7, 8, 2, 4, 6] than with a linear layer? That's what you mean by permutation equivariant? I understand each context input is embedded with its position, but I suppose the transformer can learn to ignore the position and just look at the context as an unordered set?
eachro 4y ago
What makes it approximately permutation equivariant (vs entirely)? As I understand things, if the order is jumbled, the attention matrix does get its rows and cols permuted in the way you'd expect so I'd have thought they'd be entirely permutation equivariant.
- YetAnotherNick 4y ago
  Inputs have position encoding in them.

spyder 4y ago

Yea, this paper says that the embarrassingly simple linear is better than transformers for time-series: https://arxiv.org/abs/2205.13504 https://arxiv.org/abs/2205.13504 from the abstract: "Recently, there has been a surge of Transformer-based solutions for the long-term time series forecasting (LTSF) task. Despite the growing performance over the past few years, we question the validity of this line of research in this work. Specifically, Transformers is arguably the most successful solution to extract the semantic correlations among the elements in a long sequence. However, in time series modeling, we are to extract the temporal relations in an ordered set of continuous points. While employing positional encoding and using tokens to embed sub-series in Transformers facilitate preserving some ordering information, the nature of the permutation-invariant self-attention mechanism inevitably results in temporal information loss..."