Yeah, he's glossing over some things, but with good reason. Might be more accurate to say BERT is a discriminant model, while GPT is a generative model. BERT wa

Yeah, he's glossing over some things, but with good reason. Might be more accurate to say BERT is a discriminant model, while GPT is a generative model. BERT was trained using Masked Language Model process, which is different from the decoder-only process used for the first GPT. Sentiment prediction seems like more of a particular thing BERT is capable of. There are many more capabilities, but GPT has sort of steered the industry towards generative models. https://huggingface.co/docs/transformers/main/tasks/masked_language_modeling https://huggingface.co/docs/transformers/main/tasks/masked_l... GPT and BERT were actually the first models published after Attention was published by Google.