3 ms·
The importance of the "Attention is All You Need" paper by Vaswani et al., in 2017 is that it introduced the Transformer type of model architecture. The model
by QuantumGood 3y ago
The importance of the "Attention is All You Need" paper by Vaswani et al., in 2017 is that it introduced the Transformer type of model architecture.
The model is so named because it "transforms" one sequence into another. For example, in a machine translation task, it can transform a sentence in one language into a sentence in another language.
The key innovation of the Transformer model is the use of self-attention mechanisms. This means that instead of processing the input sequence word by word, the model considers all the words in the sequence at the same time and learns to pay "attention" to the most important ones for the given task.
In essence, the Transformer model is a design for building network architectures that can process data in parallel and focus on different parts of the data depending on the task at hand. The Transformer model has proven to be highly effective and flexible, and has been adopted in many variants and applications, including BERT, GPT, T5, and many others.
- hackernewds 3y agogreat explanation. thank you all for contributing to our learning! > in a machine translation task, it transforms a sentence in one language into a sentence in another language. here English is being translated to which language - I'm assuming vectors? might be a silly question, I'm assuming that's where the origin of the word "Transformer" lies
- QuantumGood 3y ago> in a machine translation task, it can transform a sentence in one language into a sentence in another language. This means when it is translating between two human languages "a machine translation task".
- andreyk 3y agoJust a quick clarification, attention of the same sort transformers used was already being employed in RNNs for a while. Thus the name "attention is all you need", it turned out you can just remove the recurrent part which makes it hard to train the NN.
- mirekrusin 3y agoInnovation is "attention", not just "self-attention" (cross-attention for ie. translation <<encoder>>, self-attention for generation <<decoder>>). It's general computation model, does't have to work on text only. It's also general in the sense that you can mask it - ie. with lower triangular matrix so future doesn't influence past (decoder, generation); leave it unmasked (ie. in encoder, ie. in text translation you want attention to have access to full input text) or anything else really.