3 ms·
BERT is encoder only and was designed for classification and natural language inference problems. The original Transformer was encoder-decoder and was designed
by rolisz 3y ago
BERT is encoder only and was designed for classification and natural language inference problems. The original Transformer was encoder-decoder and was designed for translation.
BERT can't be used in an autoregressive way because it doesn't output a new token, it simply generates embeddings from the existing tokens (you get one for each input token).
- amilios 3y agoTo clarify for people familiar with the names but not what corresponds to what: BERT - Encoder-only - embeddings for downstream tasks GPT/OPT/etc - Decoder-only - language generation T5/T0 - Encoder-decoder. Kind of does both?