3 ms·

> gradient descent funny mentioning the math but not the Transformer encoders..

• jampekka 2y ago
  Transformer encoders are not really popular anymore, and all the top LLMs are decoder-only architectures. But encoder models like BERT are used for some tasks. In any case, self-attention and MLP is the crux of Transformer blocks, be they in the decoder or the encoder.
  • mistrial9 2y ago
    > Transformer encoders are not really popular anymore references, please