3 ms·
> it's a transformer based natural language model just like GPT its an encoder-decoder model whereas GPT is decoder only. feels like a pretty big difference, t
by swyx 3y ago
> it's a transformer based natural language model just like GPT
its an encoder-decoder model whereas GPT is decoder only. feels like a pretty big difference, though in practice i honestly still dont have a strong grasp of how encoder-decoder is deficient to decoder-only when it comes to text generation. i get that BERT was designed for translation but why cant we scale it up and use it for textgen just the same
- abhijitr 3y agoBERT is encoder only..
- rolisz 3y agoBERT is encoder only and was designed for classification and natural language inference problems. The original Transformer was encoder-decoder and was designed for translation. BERT can't be used in an autoregressive way because it doesn't output a new token, it simply generates embeddings from the existing tokens (you get one for each input token).
- amilios 3y agoTo clarify for people familiar with the names but not what corresponds to what: BERT - Encoder-only - embeddings for downstream tasks GPT/OPT/etc - Decoder-only - language generation T5/T0 - Encoder-decoder. Kind of does both?