3 ms·

Thanks. That’s helping the clear it up. So what’s the point of the encoding and decoding steps? Then the transformer is just another neural net in between tha

Thanks. That’s helping the clear it up. So what’s the point of the encoding and decoding steps? Then the transformer is just another neural net in between that does some funny business with attention?

saynay 5y ago
The transformer is the neural net in the encode / decode steps. For something like a language model, it takes an input phrase, encodes it to something that has "meaning" to the net, then decodes it into an output phrase. The "attention" mechanism learns how much attention to give to each part of the input, anywhere in it. In languages, the previous common mechanisms (recurrent networks), would only look backwards, with decreasing weight for things further away.