3 ms·
> where GPT is used to generate text, BERT is used to generate embeddings for input text that you can then use for predictive models (e.g. sentiment prediction)
by amelius 3y ago
> where GPT is used to generate text, BERT is used to generate embeddings for input text that you can then use for predictive models (e.g. sentiment prediction)
Ok, but isn't text generation more general? E.g. you could ask it to predict the sentiment of a sentence and write the result as a sentence?
- samvher 3y agoYeah my explanation was definitely a lossy summary. You can do similar things with GPT, but BERT is bidirectional, so for a given token it can take into account both tokens before and after it. GPT would only take into account tokens before it. Looking both ways can be helpful. Another comment in this thread explains the same (maybe clearer).
- extasia 3y agoBERT is ~100m parameters. GPT3.5 is 1000x bigger. A Bert sized decoder model would likely generate nonsense For most applications BERTs output is used to fine tune additional NN layers, eg for text classification
- jakderrida 3y agoYeah, he's glossing over some things, but with good reason. Might be more accurate to say BERT is a discriminant model, while GPT is a generative model. BERT was trained using Masked Language Model process, which is different from the decoder-only process used for the first GPT. Sentiment prediction seems like more of a particular thing BERT is capable of. There are many more capabilities, but GPT has sort of steered the industry towards generative models. https://huggingface.co/docs/transformers/main/tasks/masked_language_modeling https://huggingface.co/docs/transformers/main/tasks/masked_l... GPT and BERT were actually the first models published after Attention was published by Google.