Innovation is "attention", not just "self-attention" (cross-attention for ie. translation <<encoder>>, self-attention for generation <<decoder>>). It's general

Innovation is "attention", not just "self-attention" (cross-attention for ie. translation <<encoder>>, self-attention for generation <<decoder>>). It's general computation model, does't have to work on text only. It's also general in the sense that you can mask it - ie. with lower triangular matrix so future doesn't influence past (decoder, generation); leave it unmasked (ie. in encoder, ie. in text translation you want attention to have access to full input text) or anything else really.